mercurial/keepalive.py
author Matt Mackall <mpm@selenic.com>
Fri, 30 Dec 2011 17:04:36 -0600
changeset 15751 742c766c8cea
parent 14958 fd246aefedd3
child 15782 7de7630053cb
permissions -rw-r--r--
merge with stable
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     1
#   This library is free software; you can redistribute it and/or
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     2
#   modify it under the terms of the GNU Lesser General Public
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     3
#   License as published by the Free Software Foundation; either
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     4
#   version 2.1 of the License, or (at your option) any later version.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     5
#
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     6
#   This library is distributed in the hope that it will be useful,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     7
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     8
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     9
#   Lesser General Public License for more details.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    10
#
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    11
#   You should have received a copy of the GNU Lesser General Public
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
    12
#   License along with this library; if not, write to the
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
    13
#      Free Software Foundation, Inc.,
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
    14
#      59 Temple Place, Suite 330,
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    15
#      Boston, MA  02111-1307  USA
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    16
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    17
# This file is part of urlgrabber, a high-level cross-protocol url-grabber
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    18
# Copyright 2002-2004 Michael D. Stenner, Ryan Tomayko
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    19
4026
8520a773a141 fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2600
diff changeset
    20
# Modified by Benoit Boissinot:
8520a773a141 fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2600
diff changeset
    21
#  - fix for digest auth (inspired from urllib2.py @ Python v2.4)
6470
ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 6001
diff changeset
    22
# Modified by Dirkjan Ochtman:
ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 6001
diff changeset
    23
#  - import md5 function from a local util module
8296
908c5906091b util: remove md5
Martin Geisler <mg@lazybytes.net>
parents: 8233
diff changeset
    24
# Modified by Martin Geisler:
908c5906091b util: remove md5
Martin Geisler <mg@lazybytes.net>
parents: 8233
diff changeset
    25
#  - moved md5 function from local util module to this module
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
    26
# Modified by Augie Fackler:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
    27
#  - add safesend method and use it to prevent broken pipe errors
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
    28
#    on large POST requests
4026
8520a773a141 fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2600
diff changeset
    29
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    30
"""An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    31
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    32
>>> import urllib2
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    33
>>> from keepalive import HTTPHandler
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    34
>>> keepalive_handler = HTTPHandler()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    35
>>> opener = urllib2.build_opener(keepalive_handler)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    36
>>> urllib2.install_opener(opener)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
    37
>>>
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    38
>>> fo = urllib2.urlopen('http://www.python.org')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    39
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    40
If a connection to a given host is requested, and all of the existing
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    41
connections are still in use, another connection will be opened.  If
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    42
the handler tries to use an existing connection but it fails in some
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    43
way, it will be closed and removed from the pool.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    44
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    45
To remove the handler, simply re-run build_opener with no arguments, and
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    46
install that opener.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    47
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    48
You can explicitly close connections by using the close_connection()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    49
method of the returned file-like object (described below) or you can
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    50
use the handler methods:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    51
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    52
  close_connection(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    53
  close_all()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    54
  open_connections()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    55
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    56
NOTE: using the close_connection and close_all methods of the handler
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    57
should be done with care when using multiple threads.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    58
  * there is nothing that prevents another thread from creating new
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    59
    connections immediately after connections are closed
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    60
  * no checks are done to prevent in-use connections from being closed
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    61
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    62
>>> keepalive_handler.close_all()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    63
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    64
EXTRA ATTRIBUTES AND METHODS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    65
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    66
  Upon a status of 200, the object returned has a few additional
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    67
  attributes and methods, which should not be used if you want to
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    68
  remain consistent with the normal urllib2-returned objects:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    69
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    70
    close_connection()  -  close the connection to the host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    71
    readlines()         -  you know, readlines()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    72
    status              -  the return status (ie 404)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    73
    reason              -  english translation of status (ie 'File not found')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    74
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    75
  If you want the best of both worlds, use this inside an
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    76
  AttributeError-catching try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    77
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    78
  >>> try: status = fo.status
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    79
  >>> except AttributeError: status = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    80
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    81
  Unfortunately, these are ONLY there if status == 200, so it's not
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    82
  easy to distinguish between non-200 responses.  The reason is that
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    83
  urllib2 tries to do clever things with error codes 301, 302, 401,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    84
  and 407, and it wraps the object upon return.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    85
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    86
  For python versions earlier than 2.4, you can avoid this fancy error
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    87
  handling by setting the module-level global HANDLE_ERRORS to zero.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    88
  You see, prior to 2.4, it's the HTTP Handler's job to determine what
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    89
  to handle specially, and what to just pass up.  HANDLE_ERRORS == 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    90
  means "pass everything up".  In python 2.4, however, this job no
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    91
  longer belongs to the HTTP Handler and is now done by a NEW handler,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    92
  HTTPErrorProcessor.  Here's the bottom line:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    93
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    94
    python version < 2.4
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    95
        HANDLE_ERRORS == 1  (default) pass up 200, treat the rest as
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    96
                            errors
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    97
        HANDLE_ERRORS == 0  pass everything up, error processing is
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    98
                            left to the calling code
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    99
    python version >= 2.4
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   100
        HANDLE_ERRORS == 1  pass up 200, treat the rest as errors
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   101
        HANDLE_ERRORS == 0  (default) pass everything up, let the
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   102
                            other handlers (specifically,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   103
                            HTTPErrorProcessor) decide what to do
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   104
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   105
  In practice, setting the variable either way makes little difference
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   106
  in python 2.4, so for the most consistent behavior across versions,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   107
  you probably just want to use the defaults, which will give you
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   108
  exceptions on errors.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   109
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   110
"""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   111
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
   112
# $Id: keepalive.py,v 1.14 2006/04/04 21:00:32 mstenner Exp $
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   113
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   114
import errno
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   115
import httplib
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   116
import socket
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   117
import thread
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   118
import urllib2
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   119
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   120
DEBUG = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   121
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   122
import sys
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   123
if sys.version_info < (2, 4):
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   124
    HANDLE_ERRORS = 1
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   125
else: HANDLE_ERRORS = 0
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   126
14764
a7d5816087a9 classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents: 14494
diff changeset
   127
class ConnectionManager(object):
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   128
    """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   129
    The connection manager must be able to:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   130
      * keep track of all existing
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   131
      """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   132
    def __init__(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   133
        self._lock = thread.allocate_lock()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   134
        self._hostmap = {} # map hosts to a list of connections
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   135
        self._connmap = {} # map connections to host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   136
        self._readymap = {} # map connection to ready state
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   137
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   138
    def add(self, host, connection, ready):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   139
        self._lock.acquire()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   140
        try:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   141
            if not host in self._hostmap:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   142
                self._hostmap[host] = []
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   143
            self._hostmap[host].append(connection)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   144
            self._connmap[connection] = host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   145
            self._readymap[connection] = ready
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   146
        finally:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   147
            self._lock.release()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   148
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   149
    def remove(self, connection):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   150
        self._lock.acquire()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   151
        try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   152
            try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   153
                host = self._connmap[connection]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   154
            except KeyError:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   155
                pass
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   156
            else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   157
                del self._connmap[connection]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   158
                del self._readymap[connection]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   159
                self._hostmap[host].remove(connection)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   160
                if not self._hostmap[host]: del self._hostmap[host]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   161
        finally:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   162
            self._lock.release()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   163
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   164
    def set_ready(self, connection, ready):
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   165
        try:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   166
            self._readymap[connection] = ready
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   167
        except KeyError:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   168
            pass
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   169
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   170
    def get_ready_conn(self, host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   171
        conn = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   172
        self._lock.acquire()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   173
        try:
5915
d0576d065993 Prefer i in d over d.has_key(i)
Christian Ebert <blacktrash@gmx.net>
parents: 4026
diff changeset
   174
            if host in self._hostmap:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   175
                for c in self._hostmap[host]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   176
                    if self._readymap[c]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   177
                        self._readymap[c] = 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   178
                        conn = c
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   179
                        break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   180
        finally:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   181
            self._lock.release()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   182
        return conn
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   183
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   184
    def get_all(self, host=None):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   185
        if host:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   186
            return list(self._hostmap.get(host, []))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   187
        else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   188
            return dict(self._hostmap)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   189
14764
a7d5816087a9 classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents: 14494
diff changeset
   190
class KeepAliveHandler(object):
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   191
    def __init__(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   192
        self._cm = ConnectionManager()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   193
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   194
    #### Connection Management
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   195
    def open_connections(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   196
        """return a list of connected hosts and the number of connections
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   197
        to each.  [('foo.com:80', 2), ('bar.org', 1)]"""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   198
        return [(host, len(li)) for (host, li) in self._cm.get_all().items()]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   199
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   200
    def close_connection(self, host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   201
        """close connection(s) to <host>
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   202
        host is the host:port spec, as in 'www.cnn.com:8080' as passed in.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   203
        no error occurs if there is no connection to that host."""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   204
        for h in self._cm.get_all(host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   205
            self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   206
            h.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   207
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   208
    def close_all(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   209
        """close all open connections"""
7622
4dd7b28003d2 use dict.iteritems() rather than dict.items()
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 6470
diff changeset
   210
        for host, conns in self._cm.get_all().iteritems():
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   211
            for h in conns:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   212
                self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   213
                h.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   214
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   215
    def _request_closed(self, request, host, connection):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   216
        """tells us that this request is now closed and the the
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   217
        connection is ready for another request"""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   218
        self._cm.set_ready(connection, 1)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   219
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   220
    def _remove_connection(self, host, connection, close=0):
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   221
        if close:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   222
            connection.close()
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   223
        self._cm.remove(connection)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   224
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   225
    #### Transaction Execution
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   226
    def http_open(self, req):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   227
        return self.do_open(HTTPConnection, req)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   228
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   229
    def do_open(self, http_class, req):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   230
        host = req.get_host()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   231
        if not host:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   232
            raise urllib2.URLError('no host given')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   233
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   234
        try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   235
            h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   236
            while h:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   237
                r = self._reuse_connection(h, req, host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   238
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   239
                # if this response is non-None, then it worked and we're
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   240
                # done.  Break out, skipping the else block.
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   241
                if r:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   242
                    break
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   243
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   244
                # connection is bad - possibly closed by server
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   245
                # discard it and ask for the next free connection
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   246
                h.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   247
                self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   248
                h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   249
            else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   250
                # no (working) free connections were found.  Create a new one.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   251
                h = http_class(host)
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   252
                if DEBUG:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   253
                    DEBUG.info("creating new connection to %s (%d)",
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   254
                               host, id(h))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   255
                self._cm.add(host, h, 0)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   256
                self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   257
                r = h.getresponse()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   258
        except (socket.error, httplib.HTTPException), err:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   259
            raise urllib2.URLError(err)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   260
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   261
        # if not a persistent connection, don't try to reuse it
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   262
        if r.will_close:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   263
            self._cm.remove(h)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   264
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   265
        if DEBUG:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   266
            DEBUG.info("STATUS: %s, %s", r.status, r.reason)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   267
        r._handler = self
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   268
        r._host = host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   269
        r._url = req.get_full_url()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   270
        r._connection = h
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   271
        r.code = r.status
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
   272
        r.headers = r.msg
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
   273
        r.msg = r.reason
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   274
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   275
        if r.status == 200 or not HANDLE_ERRORS:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   276
            return r
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   277
        else:
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
   278
            return self.parent.error('http', req, r,
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
   279
                                     r.status, r.msg, r.headers)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   280
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   281
    def _reuse_connection(self, h, req, host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   282
        """start the transaction with a re-used connection
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   283
        return a response object (r) upon success or None on failure.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   284
        This DOES not close or remove bad connections in cases where
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   285
        it returns.  However, if an unexpected exception occurs, it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   286
        will close and remove the connection before re-raising.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   287
        """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   288
        try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   289
            self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   290
            r = h.getresponse()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   291
            # note: just because we got something back doesn't mean it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   292
            # worked.  We'll check the version below, too.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   293
        except (socket.error, httplib.HTTPException):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   294
            r = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   295
        except:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   296
            # adding this block just in case we've missed
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   297
            # something we will still raise the exception, but
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   298
            # lets try and close the connection and remove it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   299
            # first.  We previously got into a nasty loop
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   300
            # where an exception was uncaught, and so the
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   301
            # connection stayed open.  On the next try, the
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   302
            # same exception was raised, etc.  The tradeoff is
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   303
            # that it's now possible this call will raise
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   304
            # a DIFFERENT exception
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   305
            if DEBUG:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   306
                DEBUG.error("unexpected exception - closing "
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   307
                            "connection to %s (%d)", host, id(h))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   308
            self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   309
            h.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   310
            raise
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   311
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   312
        if r is None or r.version == 9:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   313
            # httplib falls back to assuming HTTP 0.9 if it gets a
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   314
            # bad header back.  This is most likely to happen if
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   315
            # the socket has been closed by the server since we
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   316
            # last used the connection.
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   317
            if DEBUG:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   318
                DEBUG.info("failed to re-use connection to %s (%d)",
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   319
                           host, id(h))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   320
            r = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   321
        else:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   322
            if DEBUG:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   323
                DEBUG.info("re-using connection to %s (%d)", host, id(h))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   324
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   325
        return r
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   326
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   327
    def _start_transaction(self, h, req):
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   328
        # What follows mostly reimplements HTTPConnection.request()
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   329
        # except it adds self.parent.addheaders in the mix.
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   330
        headers = req.headers.copy()
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   331
        if sys.version_info >= (2, 4):
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   332
            headers.update(req.unredirected_hdrs)
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   333
        headers.update(self.parent.addheaders)
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   334
        headers = dict((n.lower(), v) for n, v in headers.items())
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   335
        skipheaders = {}
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   336
        for n in ('host', 'accept-encoding'):
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   337
            if n in headers:
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   338
                skipheaders['skip_' + n.replace('-', '_')] = 1
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   339
        try:
8146
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   340
            if req.has_data():
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   341
                data = req.get_data()
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   342
                h.putrequest('POST', req.get_selector(), **skipheaders)
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   343
                if 'content-type' not in headers:
8146
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   344
                    h.putheader('Content-type',
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   345
                                'application/x-www-form-urlencoded')
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   346
                if 'content-length' not in headers:
8146
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   347
                    h.putheader('Content-length', '%d' % len(data))
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   348
            else:
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   349
                h.putrequest('GET', req.get_selector(), **skipheaders)
8146
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   350
        except (socket.error), err:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   351
            raise urllib2.URLError(err)
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   352
        for k, v in headers.items():
8146
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   353
            h.putheader(k, v)
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   354
        h.endheaders()
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   355
        if req.has_data():
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   356
            h.send(data)
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   357
5983
6f1fcbc58efa httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4026
diff changeset
   358
class HTTPHandler(KeepAliveHandler, urllib2.HTTPHandler):
6f1fcbc58efa httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4026
diff changeset
   359
    pass
6f1fcbc58efa httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4026
diff changeset
   360
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   361
class HTTPResponse(httplib.HTTPResponse):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   362
    # we need to subclass HTTPResponse in order to
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   363
    # 1) add readline() and readlines() methods
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   364
    # 2) add close_connection() methods
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   365
    # 3) add info() and geturl() methods
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   366
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   367
    # in order to add readline(), read must be modified to deal with a
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   368
    # buffer.  example: readline must read a buffer and then spit back
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   369
    # one line at a time.  The only real alternative is to read one
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   370
    # BYTE at a time (ick).  Once something has been read, it can't be
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   371
    # put back (ok, maybe it can, but that's even uglier than this),
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   372
    # so if you THEN do a normal read, you must first take stuff from
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   373
    # the buffer.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   374
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   375
    # the read method wraps the original to accomodate buffering,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   376
    # although read() never adds to the buffer.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   377
    # Both readline and readlines have been stolen with almost no
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   378
    # modification from socket.py
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   379
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   380
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   381
    def __init__(self, sock, debuglevel=0, strict=0, method=None):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   382
        if method: # the httplib in python 2.3 uses the method arg
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   383
            httplib.HTTPResponse.__init__(self, sock, debuglevel, method)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   384
        else: # 2.2 doesn't
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   385
            httplib.HTTPResponse.__init__(self, sock, debuglevel)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   386
        self.fileno = sock.fileno
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   387
        self.code = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   388
        self._rbuf = ''
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   389
        self._rbufsize = 8096
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   390
        self._handler = None # inserted by the handler later
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   391
        self._host = None    # (same)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   392
        self._url = None     # (same)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   393
        self._connection = None # (same)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   394
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   395
    _raw_read = httplib.HTTPResponse.read
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   396
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   397
    def close(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   398
        if self.fp:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   399
            self.fp.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   400
            self.fp = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   401
            if self._handler:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   402
                self._handler._request_closed(self, self._host,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   403
                                              self._connection)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   404
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   405
    def close_connection(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   406
        self._handler._remove_connection(self._host, self._connection, close=1)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   407
        self.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   408
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   409
    def info(self):
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
   410
        return self.headers
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   411
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   412
    def geturl(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   413
        return self._url
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   414
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   415
    def read(self, amt=None):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   416
        # the _rbuf test is only in this first if for speed.  It's not
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   417
        # logically necessary
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   418
        if self._rbuf and not amt is None:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   419
            L = len(self._rbuf)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   420
            if amt > L:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   421
                amt -= L
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   422
            else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   423
                s = self._rbuf[:amt]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   424
                self._rbuf = self._rbuf[amt:]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   425
                return s
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   426
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   427
        s = self._rbuf + self._raw_read(amt)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   428
        self._rbuf = ''
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   429
        return s
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   430
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   431
    # stolen from Python SVN #68532 to fix issue1088
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   432
    def _read_chunked(self, amt):
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   433
        chunk_left = self.chunk_left
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   434
        value = ''
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   435
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   436
        # XXX This accumulates chunks by repeated string concatenation,
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   437
        # which is not efficient as the number or size of chunks gets big.
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   438
        while True:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   439
            if chunk_left is None:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   440
                line = self.fp.readline()
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   441
                i = line.find(';')
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   442
                if i >= 0:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   443
                    line = line[:i] # strip chunk-extensions
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   444
                try:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   445
                    chunk_left = int(line, 16)
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   446
                except ValueError:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   447
                    # close the connection as protocol synchronisation is
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   448
                    # probably lost
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   449
                    self.close()
7868
2b901f9b37aa keepalive: fix reference to IncompleteRead
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents: 7781
diff changeset
   450
                    raise httplib.IncompleteRead(value)
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   451
                if chunk_left == 0:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   452
                    break
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   453
            if amt is None:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   454
                value += self._safe_read(chunk_left)
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   455
            elif amt < chunk_left:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   456
                value += self._safe_read(amt)
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   457
                self.chunk_left = chunk_left - amt
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   458
                return value
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   459
            elif amt == chunk_left:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   460
                value += self._safe_read(amt)
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   461
                self._safe_read(2)  # toss the CRLF at the end of the chunk
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   462
                self.chunk_left = None
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   463
                return value
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   464
            else:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   465
                value += self._safe_read(chunk_left)
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   466
                amt -= chunk_left
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   467
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   468
            # we read the whole chunk, get another
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   469
            self._safe_read(2)      # toss the CRLF at the end of the chunk
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   470
            chunk_left = None
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   471
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   472
        # read and discard trailer up to the CRLF terminator
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   473
        ### note: we shouldn't have any trailers!
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   474
        while True:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   475
            line = self.fp.readline()
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   476
            if not line:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   477
                # a vanishingly small number of sites EOF without
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   478
                # sending the trailer
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   479
                break
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   480
            if line == '\r\n':
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   481
                break
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   482
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   483
        # we read everything; close the "file"
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   484
        self.close()
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   485
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   486
        return value
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   487
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   488
    def readline(self, limit=-1):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   489
        i = self._rbuf.find('\n')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   490
        while i < 0 and not (0 < limit <= len(self._rbuf)):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   491
            new = self._raw_read(self._rbufsize)
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   492
            if not new:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   493
                break
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   494
            i = new.find('\n')
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   495
            if i >= 0:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   496
                i = i + len(self._rbuf)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   497
            self._rbuf = self._rbuf + new
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   498
        if i < 0:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   499
            i = len(self._rbuf)
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   500
        else:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   501
            i = i + 1
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   502
        if 0 <= limit < len(self._rbuf):
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   503
            i = limit
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   504
        data, self._rbuf = self._rbuf[:i], self._rbuf[i:]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   505
        return data
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   506
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   507
    def readlines(self, sizehint = 0):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   508
        total = 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   509
        list = []
14494
1ffeeb91c55d check-code: flag 0/1 used as constant Boolean expression
Martin Geisler <mg@lazybytes.net>
parents: 10394
diff changeset
   510
        while True:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   511
            line = self.readline()
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   512
            if not line:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   513
                break
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   514
            list.append(line)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   515
            total += len(line)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   516
            if sizehint and total >= sizehint:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   517
                break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   518
        return list
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   519
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   520
def safesend(self, str):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   521
    """Send `str' to the server.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   522
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   523
    Shamelessly ripped off from httplib to patch a bad behavior.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   524
    """
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   525
    # _broken_pipe_resp is an attribute we set in this function
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   526
    # if the socket is closed while we're sending data but
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   527
    # the server sent us a response before hanging up.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   528
    # In that case, we want to pretend to send the rest of the
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   529
    # outgoing data, and then let the user use getresponse()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   530
    # (which we wrap) to get this last response before
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   531
    # opening a new socket.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   532
    if getattr(self, '_broken_pipe_resp', None) is not None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   533
        return
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   534
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   535
    if self.sock is None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   536
        if self.auto_open:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   537
            self.connect()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   538
        else:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   539
            raise httplib.NotConnected()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   540
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   541
    # send the data to the server. if we get a broken pipe, then close
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   542
    # the socket. we want to reconnect when somebody tries to send again.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   543
    #
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   544
    # NOTE: we DO propagate the error, though, because we cannot simply
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   545
    #       ignore the error... the caller will know if they can retry.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   546
    if self.debuglevel > 0:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   547
        print "send:", repr(str)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   548
    try:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   549
        blocksize = 8192
14958
fd246aefedd3 keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14764
diff changeset
   550
        read = getattr(str, 'read', None)
fd246aefedd3 keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14764
diff changeset
   551
        if read is not None:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   552
            if self.debuglevel > 0:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   553
                print "sendIng a read()able"
14958
fd246aefedd3 keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14764
diff changeset
   554
            data = read(blocksize)
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   555
            while data:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   556
                self.sock.sendall(data)
14958
fd246aefedd3 keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14764
diff changeset
   557
                data = read(blocksize)
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   558
        else:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   559
            self.sock.sendall(str)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   560
    except socket.error, v:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   561
        reraise = True
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   562
        if v[0] == errno.EPIPE:      # Broken pipe
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   563
            if self._HTTPConnection__state == httplib._CS_REQ_SENT:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   564
                self._broken_pipe_resp = None
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   565
                self._broken_pipe_resp = self.getresponse()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   566
                reraise = False
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   567
            self.close()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   568
        if reraise:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   569
            raise
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   570
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   571
def wrapgetresponse(cls):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   572
    """Wraps getresponse in cls with a broken-pipe sane version.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   573
    """
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   574
    def safegetresponse(self):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   575
        # In safesend() we might set the _broken_pipe_resp
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   576
        # attribute, in which case the socket has already
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   577
        # been closed and we just need to give them the response
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   578
        # back. Otherwise, we use the normal response path.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   579
        r = getattr(self, '_broken_pipe_resp', None)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   580
        if r is not None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   581
            return r
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   582
        return cls.getresponse(self)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   583
    safegetresponse.__doc__ = cls.getresponse.__doc__
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   584
    return safegetresponse
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   585
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   586
class HTTPConnection(httplib.HTTPConnection):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   587
    # use the modified response class
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   588
    response_class = HTTPResponse
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   589
    send = safesend
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   590
    getresponse = wrapgetresponse(httplib.HTTPConnection)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   591
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   592
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   593
#########################################################################
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   594
#####   TEST FUNCTIONS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   595
#########################################################################
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   596
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   597
def error_handler(url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   598
    global HANDLE_ERRORS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   599
    orig = HANDLE_ERRORS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   600
    keepalive_handler = HTTPHandler()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   601
    opener = urllib2.build_opener(keepalive_handler)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   602
    urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   603
    pos = {0: 'off', 1: 'on'}
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   604
    for i in (0, 1):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   605
        print "  fancy error handling %s (HANDLE_ERRORS = %i)" % (pos[i], i)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   606
        HANDLE_ERRORS = i
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   607
        try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   608
            fo = urllib2.urlopen(url)
7874
d812029cda85 cleanup: drop variables for unused return values
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents: 7868
diff changeset
   609
            fo.read()
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   610
            fo.close()
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   611
            try:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   612
                status, reason = fo.status, fo.reason
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   613
            except AttributeError:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   614
                status, reason = None, None
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   615
        except IOError, e:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   616
            print "  EXCEPTION: %s" % e
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   617
            raise
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   618
        else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   619
            print "  status = %s, reason = %s" % (status, reason)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   620
    HANDLE_ERRORS = orig
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   621
    hosts = keepalive_handler.open_connections()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   622
    print "open connections:", hosts
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   623
    keepalive_handler.close_all()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   624
8296
908c5906091b util: remove md5
Martin Geisler <mg@lazybytes.net>
parents: 8233
diff changeset
   625
def md5(s):
908c5906091b util: remove md5
Martin Geisler <mg@lazybytes.net>
parents: 8233
diff changeset
   626
    try:
908c5906091b util: remove md5
Martin Geisler <mg@lazybytes.net>
parents: 8233
diff changeset
   627
        from hashlib import md5 as _md5
908c5906091b util: remove md5
Martin Geisler <mg@lazybytes.net>
parents: 8233
diff changeset
   628
    except ImportError:
908c5906091b util: remove md5
Martin Geisler <mg@lazybytes.net>
parents: 8233
diff changeset
   629
        from md5 import md5 as _md5
908c5906091b util: remove md5
Martin Geisler <mg@lazybytes.net>
parents: 8233
diff changeset
   630
    global md5
908c5906091b util: remove md5
Martin Geisler <mg@lazybytes.net>
parents: 8233
diff changeset
   631
    md5 = _md5
908c5906091b util: remove md5
Martin Geisler <mg@lazybytes.net>
parents: 8233
diff changeset
   632
    return _md5(s)
908c5906091b util: remove md5
Martin Geisler <mg@lazybytes.net>
parents: 8233
diff changeset
   633
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   634
def continuity(url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   635
    format = '%25s: %s'
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   636
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   637
    # first fetch the file with the normal http handler
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   638
    opener = urllib2.build_opener()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   639
    urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   640
    fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   641
    foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   642
    fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   643
    m = md5.new(foo)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   644
    print format % ('normal urllib', m.hexdigest())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   645
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   646
    # now install the keepalive handler and try again
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   647
    opener = urllib2.build_opener(HTTPHandler())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   648
    urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   649
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   650
    fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   651
    foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   652
    fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   653
    m = md5.new(foo)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   654
    print format % ('keepalive read', m.hexdigest())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   655
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   656
    fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   657
    foo = ''
14494
1ffeeb91c55d check-code: flag 0/1 used as constant Boolean expression
Martin Geisler <mg@lazybytes.net>
parents: 10394
diff changeset
   658
    while True:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   659
        f = fo.readline()
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   660
        if f:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   661
            foo = foo + f
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   662
        else: break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   663
    fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   664
    m = md5.new(foo)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   665
    print format % ('keepalive readline', m.hexdigest())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   666
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   667
def comp(N, url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   668
    print '  making %i connections to:\n  %s' % (N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   669
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   670
    sys.stdout.write('  first using the normal urllib handlers')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   671
    # first use normal opener
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   672
    opener = urllib2.build_opener()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   673
    urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   674
    t1 = fetch(N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   675
    print '  TIME: %.3f s' % t1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   676
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   677
    sys.stdout.write('  now using the keepalive handler       ')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   678
    # now install the keepalive handler and try again
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   679
    opener = urllib2.build_opener(HTTPHandler())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   680
    urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   681
    t2 = fetch(N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   682
    print '  TIME: %.3f s' % t2
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   683
    print '  improvement factor: %.2f' % (t1 / t2)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   684
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   685
def fetch(N, url, delay=0):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   686
    import time
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   687
    lens = []
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   688
    starttime = time.time()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   689
    for i in range(N):
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   690
        if delay and i > 0:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   691
            time.sleep(delay)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   692
        fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   693
        foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   694
        fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   695
        lens.append(len(foo))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   696
    diff = time.time() - starttime
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   697
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   698
    j = 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   699
    for i in lens[1:]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   700
        j = j + 1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   701
        if not i == lens[0]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   702
            print "WARNING: inconsistent length on read %i: %i" % (j, i)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   703
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   704
    return diff
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   705
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   706
def test_timeout(url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   707
    global DEBUG
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   708
    dbbackup = DEBUG
14764
a7d5816087a9 classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents: 14494
diff changeset
   709
    class FakeLogger(object):
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   710
        def debug(self, msg, *args):
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   711
            print msg % args
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   712
        info = warning = error = debug
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   713
    DEBUG = FakeLogger()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   714
    print "  fetching the file to establish a connection"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   715
    fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   716
    data1 = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   717
    fo.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   718
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   719
    i = 20
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   720
    print "  waiting %i seconds for the server to close the connection" % i
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   721
    while i > 0:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   722
        sys.stdout.write('\r  %2i' % i)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   723
        sys.stdout.flush()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   724
        time.sleep(1)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   725
        i -= 1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   726
    sys.stderr.write('\r')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   727
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   728
    print "  fetching the file a second time"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   729
    fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   730
    data2 = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   731
    fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   732
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   733
    if data1 == data2:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   734
        print '  data are identical'
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   735
    else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   736
        print '  ERROR: DATA DIFFER'
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   737
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   738
    DEBUG = dbbackup
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   739
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   740
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   741
def test(url, N=10):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   742
    print "checking error hander (do this on a non-200)"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   743
    try: error_handler(url)
7875
553aa0cbeab6 cleanup: drop unused assignments
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents: 7874
diff changeset
   744
    except IOError:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   745
        print "exiting - exception will prevent further tests"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   746
        sys.exit()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   747
    print
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   748
    print "performing continuity test (making sure stuff isn't corrupted)"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   749
    continuity(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   750
    print
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   751
    print "performing speed comparison"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   752
    comp(N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   753
    print
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   754
    print "performing dropped-connection check"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   755
    test_timeout(url)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   756
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   757
if __name__ == '__main__':
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   758
    import time
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   759
    import sys
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   760
    try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   761
        N = int(sys.argv[1])
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   762
        url = sys.argv[2]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   763
    except:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   764
        print "%s <integer> <url>" % sys.argv[0]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   765
    else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   766
        test(url, N)