mercurial/keepalive.py
author Pulkit Goyal <7895pulkit@gmail.com>
Fri, 29 Dec 2017 05:25:27 +0530
changeset 35582 72b91f905065
parent 35364 03112a2c9c83
child 36295 19a04ca90413
permissions -rw-r--r--
py3: use node.hex(h.digest()) instead of h.hexdigest() hashlib.sha1.hexdigest() returns str on Python 3. Differential Revision: https://phab.mercurial-scm.org/D1792
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     1
#   This library is free software; you can redistribute it and/or
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     2
#   modify it under the terms of the GNU Lesser General Public
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     3
#   License as published by the Free Software Foundation; either
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     4
#   version 2.1 of the License, or (at your option) any later version.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     5
#
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     6
#   This library is distributed in the hope that it will be useful,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     7
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     8
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     9
#   Lesser General Public License for more details.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    10
#
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    11
#   You should have received a copy of the GNU Lesser General Public
15782
7de7630053cb Remove FSF mailing address from GPL headers
Martin Geisler <mg@aragost.com>
parents: 14958
diff changeset
    12
#   License along with this library; if not, see
7de7630053cb Remove FSF mailing address from GPL headers
Martin Geisler <mg@aragost.com>
parents: 14958
diff changeset
    13
#   <http://www.gnu.org/licenses/>.
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    14
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    15
# This file is part of urlgrabber, a high-level cross-protocol url-grabber
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    16
# Copyright 2002-2004 Michael D. Stenner, Ryan Tomayko
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    17
4026
8520a773a141 fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2600
diff changeset
    18
# Modified by Benoit Boissinot:
8520a773a141 fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2600
diff changeset
    19
#  - fix for digest auth (inspired from urllib2.py @ Python v2.4)
6470
ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 6001
diff changeset
    20
# Modified by Dirkjan Ochtman:
ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 6001
diff changeset
    21
#  - import md5 function from a local util module
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
    22
# Modified by Augie Fackler:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
    23
#  - add safesend method and use it to prevent broken pipe errors
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
    24
#    on large POST requests
4026
8520a773a141 fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2600
diff changeset
    25
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    26
"""An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    27
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    28
>>> import urllib2
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    29
>>> from keepalive import HTTPHandler
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    30
>>> keepalive_handler = HTTPHandler()
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
    31
>>> opener = urlreq.buildopener(keepalive_handler)
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
    32
>>> urlreq.installopener(opener)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
    33
>>>
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
    34
>>> fo = urlreq.urlopen('http://www.python.org')
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    35
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    36
If a connection to a given host is requested, and all of the existing
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    37
connections are still in use, another connection will be opened.  If
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    38
the handler tries to use an existing connection but it fails in some
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    39
way, it will be closed and removed from the pool.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    40
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    41
To remove the handler, simply re-run build_opener with no arguments, and
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    42
install that opener.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    43
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    44
You can explicitly close connections by using the close_connection()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    45
method of the returned file-like object (described below) or you can
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    46
use the handler methods:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    47
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    48
  close_connection(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    49
  close_all()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    50
  open_connections()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    51
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    52
NOTE: using the close_connection and close_all methods of the handler
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    53
should be done with care when using multiple threads.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    54
  * there is nothing that prevents another thread from creating new
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    55
    connections immediately after connections are closed
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    56
  * no checks are done to prevent in-use connections from being closed
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    57
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    58
>>> keepalive_handler.close_all()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    59
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    60
EXTRA ATTRIBUTES AND METHODS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    61
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    62
  Upon a status of 200, the object returned has a few additional
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    63
  attributes and methods, which should not be used if you want to
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    64
  remain consistent with the normal urllib2-returned objects:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    65
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    66
    close_connection()  -  close the connection to the host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    67
    readlines()         -  you know, readlines()
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17251
diff changeset
    68
    status              -  the return status (i.e. 404)
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17251
diff changeset
    69
    reason              -  english translation of status (i.e. 'File not found')
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    70
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    71
  If you want the best of both worlds, use this inside an
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    72
  AttributeError-catching try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    73
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    74
  >>> try: status = fo.status
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    75
  >>> except AttributeError: status = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    76
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    77
  Unfortunately, these are ONLY there if status == 200, so it's not
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    78
  easy to distinguish between non-200 responses.  The reason is that
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    79
  urllib2 tries to do clever things with error codes 301, 302, 401,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    80
  and 407, and it wraps the object upon return.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    81
"""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    82
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
    83
# $Id: keepalive.py,v 1.14 2006/04/04 21:00:32 mstenner Exp $
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    84
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
    85
from __future__ import absolute_import, print_function
27507
a16489f9132d keepalive: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
    86
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
    87
import errno
29341
0d83ad967bf8 cleanup: replace uses of util.(md5|sha1|sha256|sha512) with hashlib.\1
Augie Fackler <raf@durin42.com>
parents: 28883
diff changeset
    88
import hashlib
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    89
import socket
27507
a16489f9132d keepalive: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
    90
import sys
29456
e61d384e3be4 keepalive: switch from thread to threading module
Pulkit Goyal <7895pulkit@gmail.com>
parents: 29455
diff changeset
    91
import threading
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
    92
34308
9bd003052d55 keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32668
diff changeset
    93
from .i18n import _
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
    94
from . import (
35582
72b91f905065 py3: use node.hex(h.digest()) instead of h.hexdigest()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 35364
diff changeset
    95
    node,
34427
a454123f5d94 keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents: 34331
diff changeset
    96
    pycompat,
34466
1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
    97
    urllibcompat,
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
    98
    util,
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
    99
)
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   100
29455
0c741fd6158a py3: conditionalize httplib import
Pulkit Goyal <7895pulkit@gmail.com>
parents: 29341
diff changeset
   101
httplib = util.httplib
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   102
urlerr = util.urlerr
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   103
urlreq = util.urlreq
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   104
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   105
DEBUG = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   106
14764
a7d5816087a9 classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents: 14494
diff changeset
   107
class ConnectionManager(object):
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   108
    """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   109
    The connection manager must be able to:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   110
      * keep track of all existing
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   111
      """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   112
    def __init__(self):
29456
e61d384e3be4 keepalive: switch from thread to threading module
Pulkit Goyal <7895pulkit@gmail.com>
parents: 29455
diff changeset
   113
        self._lock = threading.Lock()
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   114
        self._hostmap = {} # map hosts to a list of connections
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   115
        self._connmap = {} # map connections to host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   116
        self._readymap = {} # map connection to ready state
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   117
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   118
    def add(self, host, connection, ready):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   119
        self._lock.acquire()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   120
        try:
16686
67964cda8701 cleanup: "not x in y" -> "x not in y"
Brodie Rao <brodie@sf.io>
parents: 15782
diff changeset
   121
            if host not in self._hostmap:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   122
                self._hostmap[host] = []
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   123
            self._hostmap[host].append(connection)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   124
            self._connmap[connection] = host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   125
            self._readymap[connection] = ready
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   126
        finally:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   127
            self._lock.release()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   128
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   129
    def remove(self, connection):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   130
        self._lock.acquire()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   131
        try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   132
            try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   133
                host = self._connmap[connection]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   134
            except KeyError:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   135
                pass
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   136
            else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   137
                del self._connmap[connection]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   138
                del self._readymap[connection]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   139
                self._hostmap[host].remove(connection)
34435
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34427
diff changeset
   140
                if not self._hostmap[host]:
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34427
diff changeset
   141
                    del self._hostmap[host]
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   142
        finally:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   143
            self._lock.release()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   144
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   145
    def set_ready(self, connection, ready):
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   146
        try:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   147
            self._readymap[connection] = ready
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   148
        except KeyError:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   149
            pass
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   150
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   151
    def get_ready_conn(self, host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   152
        conn = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   153
        self._lock.acquire()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   154
        try:
5915
d0576d065993 Prefer i in d over d.has_key(i)
Christian Ebert <blacktrash@gmx.net>
parents: 4026
diff changeset
   155
            if host in self._hostmap:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   156
                for c in self._hostmap[host]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   157
                    if self._readymap[c]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   158
                        self._readymap[c] = 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   159
                        conn = c
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   160
                        break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   161
        finally:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   162
            self._lock.release()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   163
        return conn
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   164
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   165
    def get_all(self, host=None):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   166
        if host:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   167
            return list(self._hostmap.get(host, []))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   168
        else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   169
            return dict(self._hostmap)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   170
14764
a7d5816087a9 classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents: 14494
diff changeset
   171
class KeepAliveHandler(object):
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   172
    def __init__(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   173
        self._cm = ConnectionManager()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   174
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   175
    #### Connection Management
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   176
    def open_connections(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   177
        """return a list of connected hosts and the number of connections
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   178
        to each.  [('foo.com:80', 2), ('bar.org', 1)]"""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   179
        return [(host, len(li)) for (host, li) in self._cm.get_all().items()]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   180
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   181
    def close_connection(self, host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   182
        """close connection(s) to <host>
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   183
        host is the host:port spec, as in 'www.cnn.com:8080' as passed in.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   184
        no error occurs if there is no connection to that host."""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   185
        for h in self._cm.get_all(host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   186
            self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   187
            h.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   188
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   189
    def close_all(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   190
        """close all open connections"""
7622
4dd7b28003d2 use dict.iteritems() rather than dict.items()
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 6470
diff changeset
   191
        for host, conns in self._cm.get_all().iteritems():
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   192
            for h in conns:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   193
                self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   194
                h.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   195
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   196
    def _request_closed(self, request, host, connection):
17251
98166640b356 help: fix some instances of 'the the'
Mads Kiilerich <mads@kiilerich.com>
parents: 16705
diff changeset
   197
        """tells us that this request is now closed and that the
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   198
        connection is ready for another request"""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   199
        self._cm.set_ready(connection, 1)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   200
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   201
    def _remove_connection(self, host, connection, close=0):
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   202
        if close:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   203
            connection.close()
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   204
        self._cm.remove(connection)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   205
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   206
    #### Transaction Execution
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   207
    def http_open(self, req):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   208
        return self.do_open(HTTPConnection, req)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   209
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   210
    def do_open(self, http_class, req):
34466
1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
   211
        host = urllibcompat.gethost(req)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   212
        if not host:
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   213
            raise urlerr.urlerror('no host given')
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   214
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   215
        try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   216
            h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   217
            while h:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   218
                r = self._reuse_connection(h, req, host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   219
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   220
                # if this response is non-None, then it worked and we're
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   221
                # done.  Break out, skipping the else block.
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   222
                if r:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   223
                    break
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   224
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   225
                # connection is bad - possibly closed by server
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   226
                # discard it and ask for the next free connection
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   227
                h.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   228
                self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   229
                h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   230
            else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   231
                # no (working) free connections were found.  Create a new one.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   232
                h = http_class(host)
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   233
                if DEBUG:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   234
                    DEBUG.info("creating new connection to %s (%d)",
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   235
                               host, id(h))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   236
                self._cm.add(host, h, 0)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   237
                self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   238
                r = h.getresponse()
34308
9bd003052d55 keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32668
diff changeset
   239
        # The string form of BadStatusLine is the status line. Add some context
9bd003052d55 keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32668
diff changeset
   240
        # to make the error message slightly more useful.
9bd003052d55 keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents: 32668
diff changeset
   241
        except httplib.BadStatusLine as err:
34427
a454123f5d94 keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents: 34331
diff changeset
   242
            raise urlerr.urlerror(
a454123f5d94 keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents: 34331
diff changeset
   243
                _('bad HTTP status line: %s') % pycompat.sysbytes(err.line))
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 22958
diff changeset
   244
        except (socket.error, httplib.HTTPException) as err:
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   245
            raise urlerr.urlerror(err)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   246
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   247
        # if not a persistent connection, don't try to reuse it
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   248
        if r.will_close:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   249
            self._cm.remove(h)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   250
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   251
        if DEBUG:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   252
            DEBUG.info("STATUS: %s, %s", r.status, r.reason)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   253
        r._handler = self
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   254
        r._host = host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   255
        r._url = req.get_full_url()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   256
        r._connection = h
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   257
        r.code = r.status
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
   258
        r.headers = r.msg
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
   259
        r.msg = r.reason
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   260
30487
88a448a12ae8 keepalive: discard legacy Python support for error handling
Augie Fackler <augie@google.com>
parents: 30473
diff changeset
   261
        return r
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   262
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   263
    def _reuse_connection(self, h, req, host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   264
        """start the transaction with a re-used connection
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   265
        return a response object (r) upon success or None on failure.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   266
        This DOES not close or remove bad connections in cases where
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   267
        it returns.  However, if an unexpected exception occurs, it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   268
        will close and remove the connection before re-raising.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   269
        """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   270
        try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   271
            self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   272
            r = h.getresponse()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   273
            # note: just because we got something back doesn't mean it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   274
            # worked.  We'll check the version below, too.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   275
        except (socket.error, httplib.HTTPException):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   276
            r = None
16705
c2d9ef43ff6c check-code: ignore naked excepts with a "re-raise" comment
Brodie Rao <brodie@sf.io>
parents: 16688
diff changeset
   277
        except: # re-raises
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   278
            # adding this block just in case we've missed
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   279
            # something we will still raise the exception, but
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   280
            # lets try and close the connection and remove it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   281
            # first.  We previously got into a nasty loop
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   282
            # where an exception was uncaught, and so the
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   283
            # connection stayed open.  On the next try, the
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17251
diff changeset
   284
            # same exception was raised, etc.  The trade-off is
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   285
            # that it's now possible this call will raise
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   286
            # a DIFFERENT exception
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   287
            if DEBUG:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   288
                DEBUG.error("unexpected exception - closing "
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   289
                            "connection to %s (%d)", host, id(h))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   290
            self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   291
            h.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   292
            raise
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   293
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   294
        if r is None or r.version == 9:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   295
            # httplib falls back to assuming HTTP 0.9 if it gets a
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   296
            # bad header back.  This is most likely to happen if
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   297
            # the socket has been closed by the server since we
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   298
            # last used the connection.
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   299
            if DEBUG:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   300
                DEBUG.info("failed to re-use connection to %s (%d)",
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   301
                           host, id(h))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   302
            r = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   303
        else:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   304
            if DEBUG:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   305
                DEBUG.info("re-using connection to %s (%d)", host, id(h))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   306
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   307
        return r
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   308
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   309
    def _start_transaction(self, h, req):
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   310
        # What follows mostly reimplements HTTPConnection.request()
31999
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30922
diff changeset
   311
        # except it adds self.parent.addheaders in the mix and sends headers
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30922
diff changeset
   312
        # in a deterministic order (to make testing easier).
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30922
diff changeset
   313
        headers = util.sortdict(self.parent.addheaders)
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30922
diff changeset
   314
        headers.update(sorted(req.headers.items()))
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30922
diff changeset
   315
        headers.update(sorted(req.unredirected_hdrs.items()))
aa836f56c3cc keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30922
diff changeset
   316
        headers = util.sortdict((n.lower(), v) for n, v in headers.items())
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   317
        skipheaders = {}
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   318
        for n in ('host', 'accept-encoding'):
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   319
            if n in headers:
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   320
                skipheaders['skip_' + n.replace('-', '_')] = 1
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   321
        try:
34466
1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
   322
            if urllibcompat.hasdata(req):
1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
   323
                data = urllibcompat.getdata(req)
30922
1beeb5185930 keepalive: honor urllib2 style get_method overrides
John Mulligan <phlogistonjohn@asynchrono.us>
parents: 30688
diff changeset
   324
                h.putrequest(
34466
1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
   325
                    req.get_method(), urllibcompat.getselector(req),
35364
03112a2c9c83 py3: handle keyword arguments correctly in keepalive.py
Pulkit Goyal <7895pulkit@gmail.com>
parents: 34466
diff changeset
   326
                    **pycompat.strkwargs(skipheaders))
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   327
                if 'content-type' not in headers:
8146
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   328
                    h.putheader('Content-type',
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   329
                                'application/x-www-form-urlencoded')
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   330
                if 'content-length' not in headers:
8146
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   331
                    h.putheader('Content-length', '%d' % len(data))
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   332
            else:
30922
1beeb5185930 keepalive: honor urllib2 style get_method overrides
John Mulligan <phlogistonjohn@asynchrono.us>
parents: 30688
diff changeset
   333
                h.putrequest(
34466
1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
   334
                    req.get_method(), urllibcompat.getselector(req),
35364
03112a2c9c83 py3: handle keyword arguments correctly in keepalive.py
Pulkit Goyal <7895pulkit@gmail.com>
parents: 34466
diff changeset
   335
                    **pycompat.strkwargs(skipheaders))
28278
b1b22185c764 keepalive: remove useless parentheses around exception type
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27616
diff changeset
   336
        except socket.error as err:
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   337
            raise urlerr.urlerror(err)
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   338
        for k, v in headers.items():
8146
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   339
            h.putheader(k, v)
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   340
        h.endheaders()
34466
1232f7fa00c3 cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
   341
        if urllibcompat.hasdata(req):
8146
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   342
            h.send(data)
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   343
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   344
class HTTPHandler(KeepAliveHandler, urlreq.httphandler):
5983
6f1fcbc58efa httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4026
diff changeset
   345
    pass
6f1fcbc58efa httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4026
diff changeset
   346
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   347
class HTTPResponse(httplib.HTTPResponse):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   348
    # we need to subclass HTTPResponse in order to
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   349
    # 1) add readline() and readlines() methods
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   350
    # 2) add close_connection() methods
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   351
    # 3) add info() and geturl() methods
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   352
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   353
    # in order to add readline(), read must be modified to deal with a
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   354
    # buffer.  example: readline must read a buffer and then spit back
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   355
    # one line at a time.  The only real alternative is to read one
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   356
    # BYTE at a time (ick).  Once something has been read, it can't be
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   357
    # put back (ok, maybe it can, but that's even uglier than this),
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   358
    # so if you THEN do a normal read, you must first take stuff from
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   359
    # the buffer.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   360
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17251
diff changeset
   361
    # the read method wraps the original to accommodate buffering,
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   362
    # although read() never adds to the buffer.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   363
    # Both readline and readlines have been stolen with almost no
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   364
    # modification from socket.py
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   365
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   366
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   367
    def __init__(self, sock, debuglevel=0, strict=0, method=None):
34427
a454123f5d94 keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents: 34331
diff changeset
   368
        extrakw = {}
a454123f5d94 keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents: 34331
diff changeset
   369
        if not pycompat.ispy3:
35364
03112a2c9c83 py3: handle keyword arguments correctly in keepalive.py
Pulkit Goyal <7895pulkit@gmail.com>
parents: 34466
diff changeset
   370
            extrakw[r'strict'] = True
03112a2c9c83 py3: handle keyword arguments correctly in keepalive.py
Pulkit Goyal <7895pulkit@gmail.com>
parents: 34466
diff changeset
   371
            extrakw[r'buffering'] = True
32667
2806c7bbcb5e keepalive: pass the correct arguments to HTTPResponse
Kyle Lippincott <spectral@google.com>
parents: 31999
diff changeset
   372
        httplib.HTTPResponse.__init__(self, sock, debuglevel=debuglevel,
34427
a454123f5d94 keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents: 34331
diff changeset
   373
                                      method=method, **extrakw)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   374
        self.fileno = sock.fileno
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   375
        self.code = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   376
        self._rbuf = ''
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   377
        self._rbufsize = 8096
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   378
        self._handler = None # inserted by the handler later
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   379
        self._host = None    # (same)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   380
        self._url = None     # (same)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   381
        self._connection = None # (same)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   382
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   383
    _raw_read = httplib.HTTPResponse.read
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   384
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   385
    def close(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   386
        if self.fp:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   387
            self.fp.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   388
            self.fp = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   389
            if self._handler:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   390
                self._handler._request_closed(self, self._host,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   391
                                              self._connection)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   392
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   393
    def close_connection(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   394
        self._handler._remove_connection(self._host, self._connection, close=1)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   395
        self.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   396
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   397
    def info(self):
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
   398
        return self.headers
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   399
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   400
    def geturl(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   401
        return self._url
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   402
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   403
    def read(self, amt=None):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   404
        # the _rbuf test is only in this first if for speed.  It's not
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   405
        # logically necessary
34331
531332502568 style: always use `x is not None` instead of `not x is None`
Alex Gaynor <agaynor@mozilla.com>
parents: 34308
diff changeset
   406
        if self._rbuf and amt is not None:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   407
            L = len(self._rbuf)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   408
            if amt > L:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   409
                amt -= L
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   410
            else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   411
                s = self._rbuf[:amt]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   412
                self._rbuf = self._rbuf[amt:]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   413
                return s
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   414
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   415
        s = self._rbuf + self._raw_read(amt)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   416
        self._rbuf = ''
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   417
        return s
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   418
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   419
    # stolen from Python SVN #68532 to fix issue1088
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   420
    def _read_chunked(self, amt):
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   421
        chunk_left = self.chunk_left
30686
8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30487
diff changeset
   422
        parts = []
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   423
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   424
        while True:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   425
            if chunk_left is None:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   426
                line = self.fp.readline()
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   427
                i = line.find(';')
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   428
                if i >= 0:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   429
                    line = line[:i] # strip chunk-extensions
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   430
                try:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   431
                    chunk_left = int(line, 16)
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   432
                except ValueError:
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17251
diff changeset
   433
                    # close the connection as protocol synchronization is
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   434
                    # probably lost
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   435
                    self.close()
30686
8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30487
diff changeset
   436
                    raise httplib.IncompleteRead(''.join(parts))
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   437
                if chunk_left == 0:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   438
                    break
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   439
            if amt is None:
30686
8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30487
diff changeset
   440
                parts.append(self._safe_read(chunk_left))
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   441
            elif amt < chunk_left:
30686
8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30487
diff changeset
   442
                parts.append(self._safe_read(amt))
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   443
                self.chunk_left = chunk_left - amt
30686
8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30487
diff changeset
   444
                return ''.join(parts)
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   445
            elif amt == chunk_left:
30686
8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30487
diff changeset
   446
                parts.append(self._safe_read(amt))
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   447
                self._safe_read(2)  # toss the CRLF at the end of the chunk
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   448
                self.chunk_left = None
30686
8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30487
diff changeset
   449
                return ''.join(parts)
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   450
            else:
30686
8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30487
diff changeset
   451
                parts.append(self._safe_read(chunk_left))
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   452
                amt -= chunk_left
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   453
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   454
            # we read the whole chunk, get another
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   455
            self._safe_read(2)      # toss the CRLF at the end of the chunk
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   456
            chunk_left = None
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   457
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   458
        # read and discard trailer up to the CRLF terminator
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   459
        ### note: we shouldn't have any trailers!
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   460
        while True:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   461
            line = self.fp.readline()
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   462
            if not line:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   463
                # a vanishingly small number of sites EOF without
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   464
                # sending the trailer
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   465
                break
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   466
            if line == '\r\n':
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   467
                break
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   468
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   469
        # we read everything; close the "file"
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   470
        self.close()
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   471
30686
8352c42a0a0d keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30487
diff changeset
   472
        return ''.join(parts)
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   473
30687
5d06f6b73a57 keepalive: remove limit argument from readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30686
diff changeset
   474
    def readline(self):
30688
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   475
        # Fast path for a line is already available in read buffer.
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   476
        i = self._rbuf.find('\n')
30688
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   477
        if i >= 0:
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   478
            i += 1
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   479
            line = self._rbuf[:i]
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   480
            self._rbuf = self._rbuf[i:]
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   481
            return line
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   482
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   483
        # No newline in local buffer. Read until we find one.
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   484
        chunks = [self._rbuf]
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   485
        i = -1
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   486
        readsize = self._rbufsize
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   487
        while True:
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   488
            new = self._raw_read(readsize)
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   489
            if not new:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   490
                break
30688
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   491
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   492
            chunks.append(new)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   493
            i = new.find('\n')
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   494
            if i >= 0:
30688
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   495
                break
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   496
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   497
        # We either have exhausted the stream or have a newline in chunks[-1].
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   498
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   499
        # EOF
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   500
        if i == -1:
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   501
            self._rbuf = ''
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   502
            return ''.join(chunks)
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   503
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   504
        i += 1
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   505
        self._rbuf = chunks[-1][i:]
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   506
        chunks[-1] = chunks[-1][:i]
dc5b594f41e9 keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30687
diff changeset
   507
        return ''.join(chunks)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   508
19872
681f7b9213a4 check-code: check for spaces around = for named parameters
Mads Kiilerich <madski@unity3d.com>
parents: 17700
diff changeset
   509
    def readlines(self, sizehint=0):
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   510
        total = 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   511
        list = []
14494
1ffeeb91c55d check-code: flag 0/1 used as constant Boolean expression
Martin Geisler <mg@lazybytes.net>
parents: 10394
diff changeset
   512
        while True:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   513
            line = self.readline()
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   514
            if not line:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   515
                break
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   516
            list.append(line)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   517
            total += len(line)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   518
            if sizehint and total >= sizehint:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   519
                break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   520
        return list
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   521
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   522
def safesend(self, str):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   523
    """Send `str' to the server.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   524
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   525
    Shamelessly ripped off from httplib to patch a bad behavior.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   526
    """
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   527
    # _broken_pipe_resp is an attribute we set in this function
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   528
    # if the socket is closed while we're sending data but
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   529
    # the server sent us a response before hanging up.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   530
    # In that case, we want to pretend to send the rest of the
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   531
    # outgoing data, and then let the user use getresponse()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   532
    # (which we wrap) to get this last response before
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   533
    # opening a new socket.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   534
    if getattr(self, '_broken_pipe_resp', None) is not None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   535
        return
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   536
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   537
    if self.sock is None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   538
        if self.auto_open:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   539
            self.connect()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   540
        else:
16687
e34106fa0dc3 cleanup: "raise SomeException()" -> "raise SomeException"
Brodie Rao <brodie@sf.io>
parents: 16686
diff changeset
   541
            raise httplib.NotConnected
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   542
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   543
    # send the data to the server. if we get a broken pipe, then close
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   544
    # the socket. we want to reconnect when somebody tries to send again.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   545
    #
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   546
    # NOTE: we DO propagate the error, though, because we cannot simply
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   547
    #       ignore the error... the caller will know if they can retry.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   548
    if self.debuglevel > 0:
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   549
        print("send:", repr(str))
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   550
    try:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   551
        blocksize = 8192
14958
fd246aefedd3 keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14764
diff changeset
   552
        read = getattr(str, 'read', None)
fd246aefedd3 keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14764
diff changeset
   553
        if read is not None:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   554
            if self.debuglevel > 0:
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   555
                print("sending a read()able")
14958
fd246aefedd3 keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14764
diff changeset
   556
            data = read(blocksize)
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   557
            while data:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   558
                self.sock.sendall(data)
14958
fd246aefedd3 keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14764
diff changeset
   559
                data = read(blocksize)
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   560
        else:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   561
            self.sock.sendall(str)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 22958
diff changeset
   562
    except socket.error as v:
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   563
        reraise = True
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   564
        if v[0] == errno.EPIPE:      # Broken pipe
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   565
            if self._HTTPConnection__state == httplib._CS_REQ_SENT:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   566
                self._broken_pipe_resp = None
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   567
                self._broken_pipe_resp = self.getresponse()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   568
                reraise = False
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   569
            self.close()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   570
        if reraise:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   571
            raise
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   572
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   573
def wrapgetresponse(cls):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   574
    """Wraps getresponse in cls with a broken-pipe sane version.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   575
    """
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   576
    def safegetresponse(self):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   577
        # In safesend() we might set the _broken_pipe_resp
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   578
        # attribute, in which case the socket has already
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   579
        # been closed and we just need to give them the response
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   580
        # back. Otherwise, we use the normal response path.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   581
        r = getattr(self, '_broken_pipe_resp', None)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   582
        if r is not None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   583
            return r
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   584
        return cls.getresponse(self)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   585
    safegetresponse.__doc__ = cls.getresponse.__doc__
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   586
    return safegetresponse
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   587
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   588
class HTTPConnection(httplib.HTTPConnection):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   589
    # use the modified response class
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   590
    response_class = HTTPResponse
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   591
    send = safesend
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   592
    getresponse = wrapgetresponse(httplib.HTTPConnection)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   593
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   594
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   595
#########################################################################
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   596
#####   TEST FUNCTIONS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   597
#########################################################################
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   598
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   599
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   600
def continuity(url):
29341
0d83ad967bf8 cleanup: replace uses of util.(md5|sha1|sha256|sha512) with hashlib.\1
Augie Fackler <raf@durin42.com>
parents: 28883
diff changeset
   601
    md5 = hashlib.md5
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   602
    format = '%25s: %s'
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   603
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   604
    # first fetch the file with the normal http handler
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   605
    opener = urlreq.buildopener()
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   606
    urlreq.installopener(opener)
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   607
    fo = urlreq.urlopen(url)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   608
    foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   609
    fo.close()
22505
232d437af120 keepalive: fix how md5 is used
Mike Hommey <mh@glandium.org>
parents: 19872
diff changeset
   610
    m = md5(foo)
35582
72b91f905065 py3: use node.hex(h.digest()) instead of h.hexdigest()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 35364
diff changeset
   611
    print(format % ('normal urllib', node.hex(m.digest())))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   612
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   613
    # now install the keepalive handler and try again
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   614
    opener = urlreq.buildopener(HTTPHandler())
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   615
    urlreq.installopener(opener)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   616
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   617
    fo = urlreq.urlopen(url)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   618
    foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   619
    fo.close()
22505
232d437af120 keepalive: fix how md5 is used
Mike Hommey <mh@glandium.org>
parents: 19872
diff changeset
   620
    m = md5(foo)
35582
72b91f905065 py3: use node.hex(h.digest()) instead of h.hexdigest()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 35364
diff changeset
   621
    print(format % ('keepalive read', node.hex(m.digest())))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   622
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   623
    fo = urlreq.urlopen(url)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   624
    foo = ''
14494
1ffeeb91c55d check-code: flag 0/1 used as constant Boolean expression
Martin Geisler <mg@lazybytes.net>
parents: 10394
diff changeset
   625
    while True:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   626
        f = fo.readline()
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   627
        if f:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   628
            foo = foo + f
34435
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34427
diff changeset
   629
        else:
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34427
diff changeset
   630
            break
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   631
    fo.close()
22505
232d437af120 keepalive: fix how md5 is used
Mike Hommey <mh@glandium.org>
parents: 19872
diff changeset
   632
    m = md5(foo)
35582
72b91f905065 py3: use node.hex(h.digest()) instead of h.hexdigest()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 35364
diff changeset
   633
    print(format % ('keepalive readline', node.hex(m.digest())))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   634
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   635
def comp(N, url):
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   636
    print('  making %i connections to:\n  %s' % (N, url))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   637
30473
39d13b8c101d py3: bulk replace sys.stdin/out/err by util's
Yuya Nishihara <yuya@tcha.org>
parents: 30463
diff changeset
   638
    util.stdout.write('  first using the normal urllib handlers')
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   639
    # first use normal opener
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   640
    opener = urlreq.buildopener()
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   641
    urlreq.installopener(opener)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   642
    t1 = fetch(N, url)
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   643
    print('  TIME: %.3f s' % t1)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   644
30473
39d13b8c101d py3: bulk replace sys.stdin/out/err by util's
Yuya Nishihara <yuya@tcha.org>
parents: 30463
diff changeset
   645
    util.stdout.write('  now using the keepalive handler       ')
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   646
    # now install the keepalive handler and try again
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   647
    opener = urlreq.buildopener(HTTPHandler())
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   648
    urlreq.installopener(opener)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   649
    t2 = fetch(N, url)
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   650
    print('  TIME: %.3f s' % t2)
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   651
    print('  improvement factor: %.2f' % (t1 / t2))
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   652
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   653
def fetch(N, url, delay=0):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   654
    import time
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   655
    lens = []
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   656
    starttime = time.time()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   657
    for i in range(N):
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   658
        if delay and i > 0:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   659
            time.sleep(delay)
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   660
        fo = urlreq.urlopen(url)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   661
        foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   662
        fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   663
        lens.append(len(foo))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   664
    diff = time.time() - starttime
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   665
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   666
    j = 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   667
    for i in lens[1:]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   668
        j = j + 1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   669
        if not i == lens[0]:
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   670
            print("WARNING: inconsistent length on read %i: %i" % (j, i))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   671
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   672
    return diff
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   673
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   674
def test_timeout(url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   675
    global DEBUG
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   676
    dbbackup = DEBUG
14764
a7d5816087a9 classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents: 14494
diff changeset
   677
    class FakeLogger(object):
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   678
        def debug(self, msg, *args):
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   679
            print(msg % args)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   680
        info = warning = error = debug
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   681
    DEBUG = FakeLogger()
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   682
    print("  fetching the file to establish a connection")
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   683
    fo = urlreq.urlopen(url)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   684
    data1 = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   685
    fo.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   686
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   687
    i = 20
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   688
    print("  waiting %i seconds for the server to close the connection" % i)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   689
    while i > 0:
30473
39d13b8c101d py3: bulk replace sys.stdin/out/err by util's
Yuya Nishihara <yuya@tcha.org>
parents: 30463
diff changeset
   690
        util.stdout.write('\r  %2i' % i)
39d13b8c101d py3: bulk replace sys.stdin/out/err by util's
Yuya Nishihara <yuya@tcha.org>
parents: 30463
diff changeset
   691
        util.stdout.flush()
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   692
        time.sleep(1)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   693
        i -= 1
30473
39d13b8c101d py3: bulk replace sys.stdin/out/err by util's
Yuya Nishihara <yuya@tcha.org>
parents: 30463
diff changeset
   694
    util.stderr.write('\r')
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   695
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   696
    print("  fetching the file a second time")
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28278
diff changeset
   697
    fo = urlreq.urlopen(url)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   698
    data2 = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   699
    fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   700
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   701
    if data1 == data2:
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   702
        print('  data are identical')
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   703
    else:
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   704
        print('  ERROR: DATA DIFFER')
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   705
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   706
    DEBUG = dbbackup
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   707
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   708
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   709
def test(url, N=10):
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   710
    print("performing continuity test (making sure stuff isn't corrupted)")
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   711
    continuity(url)
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   712
    print('')
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   713
    print("performing speed comparison")
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   714
    comp(N, url)
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   715
    print('')
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   716
    print("performing dropped-connection check")
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   717
    test_timeout(url)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   718
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   719
if __name__ == '__main__':
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   720
    import time
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   721
    try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   722
        N = int(sys.argv[1])
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   723
        url = sys.argv[2]
16688
cfb6682961b8 cleanup: replace naked excepts with more specific ones
Brodie Rao <brodie@sf.io>
parents: 16687
diff changeset
   724
    except (IndexError, ValueError):
27616
0765d8423fbc keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27507
diff changeset
   725
        print("%s <integer> <url>" % sys.argv[0])
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   726
    else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   727
        test(url, N)