annotate mercurial/keepalive.py @ 3423:a2179e78d18b

Fix RSS URLs (closes issue396)
author Brendan Cully <brendan@kublai.com>
date Mon, 16 Oct 2006 11:18:06 -0700
parents c4325f0a9b91
children 8520a773a141
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
1 # This library is free software; you can redistribute it and/or
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
2 # modify it under the terms of the GNU Lesser General Public
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
3 # License as published by the Free Software Foundation; either
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
4 # version 2.1 of the License, or (at your option) any later version.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
5 #
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
6 # This library is distributed in the hope that it will be useful,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
7 # but WITHOUT ANY WARRANTY; without even the implied warranty of
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
8 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
9 # Lesser General Public License for more details.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
10 #
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
11 # You should have received a copy of the GNU Lesser General Public
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
12 # License along with this library; if not, write to the
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
13 # Free Software Foundation, Inc.,
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
14 # 59 Temple Place, Suite 330,
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
15 # Boston, MA 02111-1307 USA
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
16
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
17 # This file is part of urlgrabber, a high-level cross-protocol url-grabber
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
18 # Copyright 2002-2004 Michael D. Stenner, Ryan Tomayko
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
19
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
20 """An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
21
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
22 >>> import urllib2
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
23 >>> from keepalive import HTTPHandler
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
24 >>> keepalive_handler = HTTPHandler()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
25 >>> opener = urllib2.build_opener(keepalive_handler)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
26 >>> urllib2.install_opener(opener)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
27 >>>
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
28 >>> fo = urllib2.urlopen('http://www.python.org')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
29
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
30 If a connection to a given host is requested, and all of the existing
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
31 connections are still in use, another connection will be opened. If
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
32 the handler tries to use an existing connection but it fails in some
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
33 way, it will be closed and removed from the pool.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
34
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
35 To remove the handler, simply re-run build_opener with no arguments, and
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
36 install that opener.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
37
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
38 You can explicitly close connections by using the close_connection()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
39 method of the returned file-like object (described below) or you can
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
40 use the handler methods:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
41
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
42 close_connection(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
43 close_all()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
44 open_connections()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
45
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
46 NOTE: using the close_connection and close_all methods of the handler
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
47 should be done with care when using multiple threads.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
48 * there is nothing that prevents another thread from creating new
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
49 connections immediately after connections are closed
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
50 * no checks are done to prevent in-use connections from being closed
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
51
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
52 >>> keepalive_handler.close_all()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
53
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
54 EXTRA ATTRIBUTES AND METHODS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
55
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
56 Upon a status of 200, the object returned has a few additional
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
57 attributes and methods, which should not be used if you want to
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
58 remain consistent with the normal urllib2-returned objects:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
59
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
60 close_connection() - close the connection to the host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
61 readlines() - you know, readlines()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
62 status - the return status (ie 404)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
63 reason - english translation of status (ie 'File not found')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
64
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
65 If you want the best of both worlds, use this inside an
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
66 AttributeError-catching try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
67
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
68 >>> try: status = fo.status
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
69 >>> except AttributeError: status = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
70
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
71 Unfortunately, these are ONLY there if status == 200, so it's not
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
72 easy to distinguish between non-200 responses. The reason is that
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
73 urllib2 tries to do clever things with error codes 301, 302, 401,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
74 and 407, and it wraps the object upon return.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
75
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
76 For python versions earlier than 2.4, you can avoid this fancy error
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
77 handling by setting the module-level global HANDLE_ERRORS to zero.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
78 You see, prior to 2.4, it's the HTTP Handler's job to determine what
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
79 to handle specially, and what to just pass up. HANDLE_ERRORS == 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
80 means "pass everything up". In python 2.4, however, this job no
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
81 longer belongs to the HTTP Handler and is now done by a NEW handler,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
82 HTTPErrorProcessor. Here's the bottom line:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
83
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
84 python version < 2.4
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
85 HANDLE_ERRORS == 1 (default) pass up 200, treat the rest as
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
86 errors
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
87 HANDLE_ERRORS == 0 pass everything up, error processing is
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
88 left to the calling code
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
89 python version >= 2.4
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
90 HANDLE_ERRORS == 1 pass up 200, treat the rest as errors
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
91 HANDLE_ERRORS == 0 (default) pass everything up, let the
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
92 other handlers (specifically,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
93 HTTPErrorProcessor) decide what to do
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
94
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
95 In practice, setting the variable either way makes little difference
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
96 in python 2.4, so for the most consistent behavior across versions,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
97 you probably just want to use the defaults, which will give you
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
98 exceptions on errors.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
99
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
100 """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
101
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
102 # $Id: keepalive.py,v 1.14 2006/04/04 21:00:32 mstenner Exp $
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
103
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
104 import urllib2
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
105 import httplib
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
106 import socket
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
107 import thread
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
108
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
109 DEBUG = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
110
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
111 import sys
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
112 if sys.version_info < (2, 4): HANDLE_ERRORS = 1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
113 else: HANDLE_ERRORS = 0
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
114
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
115 class ConnectionManager:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
116 """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
117 The connection manager must be able to:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
118 * keep track of all existing
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
119 """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
120 def __init__(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
121 self._lock = thread.allocate_lock()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
122 self._hostmap = {} # map hosts to a list of connections
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
123 self._connmap = {} # map connections to host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
124 self._readymap = {} # map connection to ready state
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
125
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
126 def add(self, host, connection, ready):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
127 self._lock.acquire()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
128 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
129 if not self._hostmap.has_key(host): self._hostmap[host] = []
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
130 self._hostmap[host].append(connection)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
131 self._connmap[connection] = host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
132 self._readymap[connection] = ready
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
133 finally:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
134 self._lock.release()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
135
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
136 def remove(self, connection):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
137 self._lock.acquire()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
138 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
139 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
140 host = self._connmap[connection]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
141 except KeyError:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
142 pass
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
143 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
144 del self._connmap[connection]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
145 del self._readymap[connection]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
146 self._hostmap[host].remove(connection)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
147 if not self._hostmap[host]: del self._hostmap[host]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
148 finally:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
149 self._lock.release()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
150
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
151 def set_ready(self, connection, ready):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
152 try: self._readymap[connection] = ready
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
153 except KeyError: pass
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
154
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
155 def get_ready_conn(self, host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
156 conn = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
157 self._lock.acquire()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
158 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
159 if self._hostmap.has_key(host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
160 for c in self._hostmap[host]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
161 if self._readymap[c]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
162 self._readymap[c] = 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
163 conn = c
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
164 break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
165 finally:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
166 self._lock.release()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
167 return conn
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
168
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
169 def get_all(self, host=None):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
170 if host:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
171 return list(self._hostmap.get(host, []))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
172 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
173 return dict(self._hostmap)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
174
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
175 class HTTPHandler(urllib2.HTTPHandler):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
176 def __init__(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
177 self._cm = ConnectionManager()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
178
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
179 #### Connection Management
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
180 def open_connections(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
181 """return a list of connected hosts and the number of connections
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
182 to each. [('foo.com:80', 2), ('bar.org', 1)]"""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
183 return [(host, len(li)) for (host, li) in self._cm.get_all().items()]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
184
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
185 def close_connection(self, host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
186 """close connection(s) to <host>
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
187 host is the host:port spec, as in 'www.cnn.com:8080' as passed in.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
188 no error occurs if there is no connection to that host."""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
189 for h in self._cm.get_all(host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
190 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
191 h.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
192
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
193 def close_all(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
194 """close all open connections"""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
195 for host, conns in self._cm.get_all().items():
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
196 for h in conns:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
197 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
198 h.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
199
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
200 def _request_closed(self, request, host, connection):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
201 """tells us that this request is now closed and the the
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
202 connection is ready for another request"""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
203 self._cm.set_ready(connection, 1)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
204
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
205 def _remove_connection(self, host, connection, close=0):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
206 if close: connection.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
207 self._cm.remove(connection)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
208
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
209 #### Transaction Execution
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
210 def http_open(self, req):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
211 return self.do_open(HTTPConnection, req)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
212
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
213 def do_open(self, http_class, req):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
214 host = req.get_host()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
215 if not host:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
216 raise urllib2.URLError('no host given')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
217
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
218 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
219 h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
220 while h:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
221 r = self._reuse_connection(h, req, host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
222
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
223 # if this response is non-None, then it worked and we're
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
224 # done. Break out, skipping the else block.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
225 if r: break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
226
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
227 # connection is bad - possibly closed by server
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
228 # discard it and ask for the next free connection
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
229 h.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
230 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
231 h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
232 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
233 # no (working) free connections were found. Create a new one.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
234 h = http_class(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
235 if DEBUG: DEBUG.info("creating new connection to %s (%d)",
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
236 host, id(h))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
237 self._cm.add(host, h, 0)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
238 self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
239 r = h.getresponse()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
240 except (socket.error, httplib.HTTPException), err:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
241 raise urllib2.URLError(err)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
242
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
243 # if not a persistent connection, don't try to reuse it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
244 if r.will_close: self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
245
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
246 if DEBUG: DEBUG.info("STATUS: %s, %s", r.status, r.reason)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
247 r._handler = self
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
248 r._host = host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
249 r._url = req.get_full_url()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
250 r._connection = h
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
251 r.code = r.status
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
252 r.headers = r.msg
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
253 r.msg = r.reason
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
254
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
255 if r.status == 200 or not HANDLE_ERRORS:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
256 return r
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
257 else:
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
258 return self.parent.error('http', req, r,
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
259 r.status, r.msg, r.headers)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
260
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
261 def _reuse_connection(self, h, req, host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
262 """start the transaction with a re-used connection
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
263 return a response object (r) upon success or None on failure.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
264 This DOES not close or remove bad connections in cases where
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
265 it returns. However, if an unexpected exception occurs, it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
266 will close and remove the connection before re-raising.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
267 """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
268 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
269 self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
270 r = h.getresponse()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
271 # note: just because we got something back doesn't mean it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
272 # worked. We'll check the version below, too.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
273 except (socket.error, httplib.HTTPException):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
274 r = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
275 except:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
276 # adding this block just in case we've missed
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
277 # something we will still raise the exception, but
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
278 # lets try and close the connection and remove it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
279 # first. We previously got into a nasty loop
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
280 # where an exception was uncaught, and so the
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
281 # connection stayed open. On the next try, the
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
282 # same exception was raised, etc. The tradeoff is
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
283 # that it's now possible this call will raise
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
284 # a DIFFERENT exception
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
285 if DEBUG: DEBUG.error("unexpected exception - closing " + \
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
286 "connection to %s (%d)", host, id(h))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
287 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
288 h.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
289 raise
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
290
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
291 if r is None or r.version == 9:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
292 # httplib falls back to assuming HTTP 0.9 if it gets a
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
293 # bad header back. This is most likely to happen if
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
294 # the socket has been closed by the server since we
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
295 # last used the connection.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
296 if DEBUG: DEBUG.info("failed to re-use connection to %s (%d)",
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
297 host, id(h))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
298 r = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
299 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
300 if DEBUG: DEBUG.info("re-using connection to %s (%d)", host, id(h))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
301
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
302 return r
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
303
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
304 def _start_transaction(self, h, req):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
305 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
306 if req.has_data():
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
307 data = req.get_data()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
308 h.putrequest('POST', req.get_selector())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
309 if not req.headers.has_key('Content-type'):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
310 h.putheader('Content-type',
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
311 'application/x-www-form-urlencoded')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
312 if not req.headers.has_key('Content-length'):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
313 h.putheader('Content-length', '%d' % len(data))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
314 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
315 h.putrequest('GET', req.get_selector())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
316 except (socket.error, httplib.HTTPException), err:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
317 raise urllib2.URLError(err)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
318
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
319 for args in self.parent.addheaders:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
320 h.putheader(*args)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
321 for k, v in req.headers.items():
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
322 h.putheader(k, v)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
323 h.endheaders()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
324 if req.has_data():
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
325 h.send(data)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
326
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
327 class HTTPResponse(httplib.HTTPResponse):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
328 # we need to subclass HTTPResponse in order to
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
329 # 1) add readline() and readlines() methods
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
330 # 2) add close_connection() methods
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
331 # 3) add info() and geturl() methods
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
332
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
333 # in order to add readline(), read must be modified to deal with a
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
334 # buffer. example: readline must read a buffer and then spit back
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
335 # one line at a time. The only real alternative is to read one
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
336 # BYTE at a time (ick). Once something has been read, it can't be
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
337 # put back (ok, maybe it can, but that's even uglier than this),
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
338 # so if you THEN do a normal read, you must first take stuff from
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
339 # the buffer.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
340
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
341 # the read method wraps the original to accomodate buffering,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
342 # although read() never adds to the buffer.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
343 # Both readline and readlines have been stolen with almost no
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
344 # modification from socket.py
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
345
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
346
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
347 def __init__(self, sock, debuglevel=0, strict=0, method=None):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
348 if method: # the httplib in python 2.3 uses the method arg
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
349 httplib.HTTPResponse.__init__(self, sock, debuglevel, method)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
350 else: # 2.2 doesn't
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
351 httplib.HTTPResponse.__init__(self, sock, debuglevel)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
352 self.fileno = sock.fileno
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
353 self.code = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
354 self._rbuf = ''
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
355 self._rbufsize = 8096
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
356 self._handler = None # inserted by the handler later
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
357 self._host = None # (same)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
358 self._url = None # (same)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
359 self._connection = None # (same)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
360
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
361 _raw_read = httplib.HTTPResponse.read
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
362
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
363 def close(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
364 if self.fp:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
365 self.fp.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
366 self.fp = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
367 if self._handler:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
368 self._handler._request_closed(self, self._host,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
369 self._connection)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
370
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
371 def close_connection(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
372 self._handler._remove_connection(self._host, self._connection, close=1)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
373 self.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
374
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
375 def info(self):
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
376 return self.headers
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
377
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
378 def geturl(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
379 return self._url
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
380
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
381 def read(self, amt=None):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
382 # the _rbuf test is only in this first if for speed. It's not
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
383 # logically necessary
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
384 if self._rbuf and not amt is None:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
385 L = len(self._rbuf)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
386 if amt > L:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
387 amt -= L
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
388 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
389 s = self._rbuf[:amt]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
390 self._rbuf = self._rbuf[amt:]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
391 return s
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
392
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
393 s = self._rbuf + self._raw_read(amt)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
394 self._rbuf = ''
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
395 return s
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
396
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
397 def readline(self, limit=-1):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
398 data = ""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
399 i = self._rbuf.find('\n')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
400 while i < 0 and not (0 < limit <= len(self._rbuf)):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
401 new = self._raw_read(self._rbufsize)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
402 if not new: break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
403 i = new.find('\n')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
404 if i >= 0: i = i + len(self._rbuf)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
405 self._rbuf = self._rbuf + new
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
406 if i < 0: i = len(self._rbuf)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
407 else: i = i+1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
408 if 0 <= limit < len(self._rbuf): i = limit
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
409 data, self._rbuf = self._rbuf[:i], self._rbuf[i:]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
410 return data
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
411
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
412 def readlines(self, sizehint = 0):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
413 total = 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
414 list = []
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
415 while 1:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
416 line = self.readline()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
417 if not line: break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
418 list.append(line)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
419 total += len(line)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
420 if sizehint and total >= sizehint:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
421 break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
422 return list
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
423
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
424
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
425 class HTTPConnection(httplib.HTTPConnection):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
426 # use the modified response class
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
427 response_class = HTTPResponse
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
428
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
429 #########################################################################
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
430 ##### TEST FUNCTIONS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
431 #########################################################################
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
432
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
433 def error_handler(url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
434 global HANDLE_ERRORS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
435 orig = HANDLE_ERRORS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
436 keepalive_handler = HTTPHandler()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
437 opener = urllib2.build_opener(keepalive_handler)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
438 urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
439 pos = {0: 'off', 1: 'on'}
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
440 for i in (0, 1):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
441 print " fancy error handling %s (HANDLE_ERRORS = %i)" % (pos[i], i)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
442 HANDLE_ERRORS = i
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
443 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
444 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
445 foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
446 fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
447 try: status, reason = fo.status, fo.reason
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
448 except AttributeError: status, reason = None, None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
449 except IOError, e:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
450 print " EXCEPTION: %s" % e
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
451 raise
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
452 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
453 print " status = %s, reason = %s" % (status, reason)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
454 HANDLE_ERRORS = orig
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
455 hosts = keepalive_handler.open_connections()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
456 print "open connections:", hosts
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
457 keepalive_handler.close_all()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
458
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
459 def continuity(url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
460 import md5
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
461 format = '%25s: %s'
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
462
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
463 # first fetch the file with the normal http handler
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
464 opener = urllib2.build_opener()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
465 urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
466 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
467 foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
468 fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
469 m = md5.new(foo)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
470 print format % ('normal urllib', m.hexdigest())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
471
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
472 # now install the keepalive handler and try again
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
473 opener = urllib2.build_opener(HTTPHandler())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
474 urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
475
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
476 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
477 foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
478 fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
479 m = md5.new(foo)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
480 print format % ('keepalive read', m.hexdigest())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
481
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
482 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
483 foo = ''
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
484 while 1:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
485 f = fo.readline()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
486 if f: foo = foo + f
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
487 else: break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
488 fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
489 m = md5.new(foo)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
490 print format % ('keepalive readline', m.hexdigest())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
491
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
492 def comp(N, url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
493 print ' making %i connections to:\n %s' % (N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
494
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
495 sys.stdout.write(' first using the normal urllib handlers')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
496 # first use normal opener
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
497 opener = urllib2.build_opener()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
498 urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
499 t1 = fetch(N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
500 print ' TIME: %.3f s' % t1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
501
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
502 sys.stdout.write(' now using the keepalive handler ')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
503 # now install the keepalive handler and try again
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
504 opener = urllib2.build_opener(HTTPHandler())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
505 urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
506 t2 = fetch(N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
507 print ' TIME: %.3f s' % t2
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
508 print ' improvement factor: %.2f' % (t1/t2, )
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
509
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
510 def fetch(N, url, delay=0):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
511 import time
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
512 lens = []
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
513 starttime = time.time()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
514 for i in range(N):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
515 if delay and i > 0: time.sleep(delay)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
516 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
517 foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
518 fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
519 lens.append(len(foo))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
520 diff = time.time() - starttime
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
521
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
522 j = 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
523 for i in lens[1:]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
524 j = j + 1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
525 if not i == lens[0]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
526 print "WARNING: inconsistent length on read %i: %i" % (j, i)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
527
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
528 return diff
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
529
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
530 def test_timeout(url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
531 global DEBUG
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
532 dbbackup = DEBUG
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
533 class FakeLogger:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
534 def debug(self, msg, *args): print msg % args
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
535 info = warning = error = debug
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
536 DEBUG = FakeLogger()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
537 print " fetching the file to establish a connection"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
538 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
539 data1 = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
540 fo.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
541
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
542 i = 20
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
543 print " waiting %i seconds for the server to close the connection" % i
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
544 while i > 0:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
545 sys.stdout.write('\r %2i' % i)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
546 sys.stdout.flush()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
547 time.sleep(1)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
548 i -= 1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
549 sys.stderr.write('\r')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
550
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
551 print " fetching the file a second time"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
552 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
553 data2 = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
554 fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
555
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
556 if data1 == data2:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
557 print ' data are identical'
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
558 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
559 print ' ERROR: DATA DIFFER'
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
560
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
561 DEBUG = dbbackup
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
562
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
563
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
564 def test(url, N=10):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
565 print "checking error hander (do this on a non-200)"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
566 try: error_handler(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
567 except IOError, e:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
568 print "exiting - exception will prevent further tests"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
569 sys.exit()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
570 print
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
571 print "performing continuity test (making sure stuff isn't corrupted)"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
572 continuity(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
573 print
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
574 print "performing speed comparison"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
575 comp(N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
576 print
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
577 print "performing dropped-connection check"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
578 test_timeout(url)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
579
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
580 if __name__ == '__main__':
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
581 import time
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
582 import sys
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
583 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
584 N = int(sys.argv[1])
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
585 url = sys.argv[2]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
586 except:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
587 print "%s <integer> <url>" % sys.argv[0]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
588 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
589 test(url, N)