Mercurial > hg
annotate mercurial/keepalive.py @ 6858:8f256bf98219
Add support for multiple possible bisect results (issue1228, issue1182)
The real reason for both issue is that bisect can not handle cases where there
are multiple possibilities for the result.
Example (from issue1228):
rev 0 -> good
rev 1 -> skipped
rev 2 -> skipped
rev 3 -> skipped
rev 4 -> bad
Note that this patch does not only fix the reported Assertion Error but also
the problem of a non converging bisect:
hg init
for i in `seq 3`; do echo $i > $i; hg add $i; hg ci -m$i; done
hg bisect -b 2
hg bisect -g 0
hg bisect -s
From this state on, you can:
a) mark as bad forever (non converging!)
b) mark as good to get an inconsistent state
c) skip for the Assertion Error
Minor description and code edits by pmezard.
author | Bernhard Leiner <bleiner@gmail.com> |
---|---|
date | Sat, 02 Aug 2008 22:10:10 +0200 |
parents | 30d2fecaab76 |
children | ac0bcd951c2c |
rev | line source |
---|---|
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
1 # This library is free software; you can redistribute it and/or |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
2 # modify it under the terms of the GNU Lesser General Public |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
3 # License as published by the Free Software Foundation; either |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
4 # version 2.1 of the License, or (at your option) any later version. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
5 # |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
6 # This library is distributed in the hope that it will be useful, |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
7 # but WITHOUT ANY WARRANTY; without even the implied warranty of |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
8 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
9 # Lesser General Public License for more details. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
10 # |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
11 # You should have received a copy of the GNU Lesser General Public |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
12 # License along with this library; if not, write to the |
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
13 # Free Software Foundation, Inc., |
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
14 # 59 Temple Place, Suite 330, |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
15 # Boston, MA 02111-1307 USA |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
16 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
17 # This file is part of urlgrabber, a high-level cross-protocol url-grabber |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
18 # Copyright 2002-2004 Michael D. Stenner, Ryan Tomayko |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
19 |
4026
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
20 # Modified by Benoit Boissinot: |
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
21 # - fix for digest auth (inspired from urllib2.py @ Python v2.4) |
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
22 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
23 """An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
24 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
25 >>> import urllib2 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
26 >>> from keepalive import HTTPHandler |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
27 >>> keepalive_handler = HTTPHandler() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
28 >>> opener = urllib2.build_opener(keepalive_handler) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
29 >>> urllib2.install_opener(opener) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
30 >>> |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
31 >>> fo = urllib2.urlopen('http://www.python.org') |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
32 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
33 If a connection to a given host is requested, and all of the existing |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
34 connections are still in use, another connection will be opened. If |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
35 the handler tries to use an existing connection but it fails in some |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
36 way, it will be closed and removed from the pool. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
37 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
38 To remove the handler, simply re-run build_opener with no arguments, and |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
39 install that opener. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
40 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
41 You can explicitly close connections by using the close_connection() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
42 method of the returned file-like object (described below) or you can |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
43 use the handler methods: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
44 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
45 close_connection(host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
46 close_all() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
47 open_connections() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
48 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
49 NOTE: using the close_connection and close_all methods of the handler |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
50 should be done with care when using multiple threads. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
51 * there is nothing that prevents another thread from creating new |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
52 connections immediately after connections are closed |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
53 * no checks are done to prevent in-use connections from being closed |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
54 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
55 >>> keepalive_handler.close_all() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
56 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
57 EXTRA ATTRIBUTES AND METHODS |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
58 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
59 Upon a status of 200, the object returned has a few additional |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
60 attributes and methods, which should not be used if you want to |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
61 remain consistent with the normal urllib2-returned objects: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
62 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
63 close_connection() - close the connection to the host |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
64 readlines() - you know, readlines() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
65 status - the return status (ie 404) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
66 reason - english translation of status (ie 'File not found') |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
67 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
68 If you want the best of both worlds, use this inside an |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
69 AttributeError-catching try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
70 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
71 >>> try: status = fo.status |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
72 >>> except AttributeError: status = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
73 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
74 Unfortunately, these are ONLY there if status == 200, so it's not |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
75 easy to distinguish between non-200 responses. The reason is that |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
76 urllib2 tries to do clever things with error codes 301, 302, 401, |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
77 and 407, and it wraps the object upon return. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
78 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
79 For python versions earlier than 2.4, you can avoid this fancy error |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
80 handling by setting the module-level global HANDLE_ERRORS to zero. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
81 You see, prior to 2.4, it's the HTTP Handler's job to determine what |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
82 to handle specially, and what to just pass up. HANDLE_ERRORS == 0 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
83 means "pass everything up". In python 2.4, however, this job no |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
84 longer belongs to the HTTP Handler and is now done by a NEW handler, |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
85 HTTPErrorProcessor. Here's the bottom line: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
86 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
87 python version < 2.4 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
88 HANDLE_ERRORS == 1 (default) pass up 200, treat the rest as |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
89 errors |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
90 HANDLE_ERRORS == 0 pass everything up, error processing is |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
91 left to the calling code |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
92 python version >= 2.4 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
93 HANDLE_ERRORS == 1 pass up 200, treat the rest as errors |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
94 HANDLE_ERRORS == 0 (default) pass everything up, let the |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
95 other handlers (specifically, |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
96 HTTPErrorProcessor) decide what to do |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
97 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
98 In practice, setting the variable either way makes little difference |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
99 in python 2.4, so for the most consistent behavior across versions, |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
100 you probably just want to use the defaults, which will give you |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
101 exceptions on errors. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
102 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
103 """ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
104 |
2444
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
105 # $Id: keepalive.py,v 1.14 2006/04/04 21:00:32 mstenner Exp $ |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
106 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
107 import urllib2 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
108 import httplib |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
109 import socket |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
110 import thread |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
111 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
112 DEBUG = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
113 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
114 import sys |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
115 if sys.version_info < (2, 4): HANDLE_ERRORS = 1 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
116 else: HANDLE_ERRORS = 0 |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
117 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
118 class ConnectionManager: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
119 """ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
120 The connection manager must be able to: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
121 * keep track of all existing |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
122 """ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
123 def __init__(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
124 self._lock = thread.allocate_lock() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
125 self._hostmap = {} # map hosts to a list of connections |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
126 self._connmap = {} # map connections to host |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
127 self._readymap = {} # map connection to ready state |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
128 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
129 def add(self, host, connection, ready): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
130 self._lock.acquire() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
131 try: |
5915
d0576d065993
Prefer i in d over d.has_key(i)
Christian Ebert <blacktrash@gmx.net>
parents:
4026
diff
changeset
|
132 if not host in self._hostmap: self._hostmap[host] = [] |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
133 self._hostmap[host].append(connection) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
134 self._connmap[connection] = host |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
135 self._readymap[connection] = ready |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
136 finally: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
137 self._lock.release() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
138 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
139 def remove(self, connection): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
140 self._lock.acquire() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
141 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
142 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
143 host = self._connmap[connection] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
144 except KeyError: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
145 pass |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
146 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
147 del self._connmap[connection] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
148 del self._readymap[connection] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
149 self._hostmap[host].remove(connection) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
150 if not self._hostmap[host]: del self._hostmap[host] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
151 finally: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
152 self._lock.release() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
153 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
154 def set_ready(self, connection, ready): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
155 try: self._readymap[connection] = ready |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
156 except KeyError: pass |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
157 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
158 def get_ready_conn(self, host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
159 conn = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
160 self._lock.acquire() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
161 try: |
5915
d0576d065993
Prefer i in d over d.has_key(i)
Christian Ebert <blacktrash@gmx.net>
parents:
4026
diff
changeset
|
162 if host in self._hostmap: |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
163 for c in self._hostmap[host]: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
164 if self._readymap[c]: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
165 self._readymap[c] = 0 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
166 conn = c |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
167 break |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
168 finally: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
169 self._lock.release() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
170 return conn |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
171 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
172 def get_all(self, host=None): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
173 if host: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
174 return list(self._hostmap.get(host, [])) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
175 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
176 return dict(self._hostmap) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
177 |
5983
6f1fcbc58efa
httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
4026
diff
changeset
|
178 class KeepAliveHandler: |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
179 def __init__(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
180 self._cm = ConnectionManager() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
181 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
182 #### Connection Management |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
183 def open_connections(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
184 """return a list of connected hosts and the number of connections |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
185 to each. [('foo.com:80', 2), ('bar.org', 1)]""" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
186 return [(host, len(li)) for (host, li) in self._cm.get_all().items()] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
187 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
188 def close_connection(self, host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
189 """close connection(s) to <host> |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
190 host is the host:port spec, as in 'www.cnn.com:8080' as passed in. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
191 no error occurs if there is no connection to that host.""" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
192 for h in self._cm.get_all(host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
193 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
194 h.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
195 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
196 def close_all(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
197 """close all open connections""" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
198 for host, conns in self._cm.get_all().items(): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
199 for h in conns: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
200 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
201 h.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
202 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
203 def _request_closed(self, request, host, connection): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
204 """tells us that this request is now closed and the the |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
205 connection is ready for another request""" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
206 self._cm.set_ready(connection, 1) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
207 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
208 def _remove_connection(self, host, connection, close=0): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
209 if close: connection.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
210 self._cm.remove(connection) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
211 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
212 #### Transaction Execution |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
213 def http_open(self, req): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
214 return self.do_open(HTTPConnection, req) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
215 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
216 def do_open(self, http_class, req): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
217 host = req.get_host() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
218 if not host: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
219 raise urllib2.URLError('no host given') |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
220 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
221 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
222 h = self._cm.get_ready_conn(host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
223 while h: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
224 r = self._reuse_connection(h, req, host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
225 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
226 # if this response is non-None, then it worked and we're |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
227 # done. Break out, skipping the else block. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
228 if r: break |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
229 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
230 # connection is bad - possibly closed by server |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
231 # discard it and ask for the next free connection |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
232 h.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
233 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
234 h = self._cm.get_ready_conn(host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
235 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
236 # no (working) free connections were found. Create a new one. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
237 h = http_class(host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
238 if DEBUG: DEBUG.info("creating new connection to %s (%d)", |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
239 host, id(h)) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
240 self._cm.add(host, h, 0) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
241 self._start_transaction(h, req) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
242 r = h.getresponse() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
243 except (socket.error, httplib.HTTPException), err: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
244 raise urllib2.URLError(err) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
245 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
246 # if not a persistent connection, don't try to reuse it |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
247 if r.will_close: self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
248 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
249 if DEBUG: DEBUG.info("STATUS: %s, %s", r.status, r.reason) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
250 r._handler = self |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
251 r._host = host |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
252 r._url = req.get_full_url() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
253 r._connection = h |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
254 r.code = r.status |
2444
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
255 r.headers = r.msg |
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
256 r.msg = r.reason |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
257 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
258 if r.status == 200 or not HANDLE_ERRORS: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
259 return r |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
260 else: |
2444
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
261 return self.parent.error('http', req, r, |
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
262 r.status, r.msg, r.headers) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
263 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
264 def _reuse_connection(self, h, req, host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
265 """start the transaction with a re-used connection |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
266 return a response object (r) upon success or None on failure. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
267 This DOES not close or remove bad connections in cases where |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
268 it returns. However, if an unexpected exception occurs, it |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
269 will close and remove the connection before re-raising. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
270 """ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
271 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
272 self._start_transaction(h, req) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
273 r = h.getresponse() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
274 # note: just because we got something back doesn't mean it |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
275 # worked. We'll check the version below, too. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
276 except (socket.error, httplib.HTTPException): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
277 r = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
278 except: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
279 # adding this block just in case we've missed |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
280 # something we will still raise the exception, but |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
281 # lets try and close the connection and remove it |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
282 # first. We previously got into a nasty loop |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
283 # where an exception was uncaught, and so the |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
284 # connection stayed open. On the next try, the |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
285 # same exception was raised, etc. The tradeoff is |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
286 # that it's now possible this call will raise |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
287 # a DIFFERENT exception |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
288 if DEBUG: DEBUG.error("unexpected exception - closing " + \ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
289 "connection to %s (%d)", host, id(h)) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
290 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
291 h.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
292 raise |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
293 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
294 if r is None or r.version == 9: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
295 # httplib falls back to assuming HTTP 0.9 if it gets a |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
296 # bad header back. This is most likely to happen if |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
297 # the socket has been closed by the server since we |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
298 # last used the connection. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
299 if DEBUG: DEBUG.info("failed to re-use connection to %s (%d)", |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
300 host, id(h)) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
301 r = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
302 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
303 if DEBUG: DEBUG.info("re-using connection to %s (%d)", host, id(h)) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
304 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
305 return r |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
306 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
307 def _start_transaction(self, h, req): |
4026
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
308 headers = req.headers.copy() |
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
309 body = req.data |
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
310 if sys.version_info >= (2, 4): |
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
311 headers.update(req.unredirected_hdrs) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
312 try: |
4026
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
313 h.request(req.get_method(), req.get_selector(), body, headers) |
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
314 except socket.error, err: # XXX what error? |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
315 raise urllib2.URLError(err) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
316 |
5983
6f1fcbc58efa
httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
4026
diff
changeset
|
317 class HTTPHandler(KeepAliveHandler, urllib2.HTTPHandler): |
6f1fcbc58efa
httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
4026
diff
changeset
|
318 pass |
6f1fcbc58efa
httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
4026
diff
changeset
|
319 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
320 class HTTPResponse(httplib.HTTPResponse): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
321 # we need to subclass HTTPResponse in order to |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
322 # 1) add readline() and readlines() methods |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
323 # 2) add close_connection() methods |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
324 # 3) add info() and geturl() methods |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
325 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
326 # in order to add readline(), read must be modified to deal with a |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
327 # buffer. example: readline must read a buffer and then spit back |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
328 # one line at a time. The only real alternative is to read one |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
329 # BYTE at a time (ick). Once something has been read, it can't be |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
330 # put back (ok, maybe it can, but that's even uglier than this), |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
331 # so if you THEN do a normal read, you must first take stuff from |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
332 # the buffer. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
333 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
334 # the read method wraps the original to accomodate buffering, |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
335 # although read() never adds to the buffer. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
336 # Both readline and readlines have been stolen with almost no |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
337 # modification from socket.py |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
338 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
339 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
340 def __init__(self, sock, debuglevel=0, strict=0, method=None): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
341 if method: # the httplib in python 2.3 uses the method arg |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
342 httplib.HTTPResponse.__init__(self, sock, debuglevel, method) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
343 else: # 2.2 doesn't |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
344 httplib.HTTPResponse.__init__(self, sock, debuglevel) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
345 self.fileno = sock.fileno |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
346 self.code = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
347 self._rbuf = '' |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
348 self._rbufsize = 8096 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
349 self._handler = None # inserted by the handler later |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
350 self._host = None # (same) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
351 self._url = None # (same) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
352 self._connection = None # (same) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
353 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
354 _raw_read = httplib.HTTPResponse.read |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
355 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
356 def close(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
357 if self.fp: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
358 self.fp.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
359 self.fp = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
360 if self._handler: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
361 self._handler._request_closed(self, self._host, |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
362 self._connection) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
363 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
364 def close_connection(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
365 self._handler._remove_connection(self._host, self._connection, close=1) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
366 self.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
367 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
368 def info(self): |
2444
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
369 return self.headers |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
370 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
371 def geturl(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
372 return self._url |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
373 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
374 def read(self, amt=None): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
375 # the _rbuf test is only in this first if for speed. It's not |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
376 # logically necessary |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
377 if self._rbuf and not amt is None: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
378 L = len(self._rbuf) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
379 if amt > L: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
380 amt -= L |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
381 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
382 s = self._rbuf[:amt] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
383 self._rbuf = self._rbuf[amt:] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
384 return s |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
385 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
386 s = self._rbuf + self._raw_read(amt) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
387 self._rbuf = '' |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
388 return s |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
389 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
390 def readline(self, limit=-1): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
391 data = "" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
392 i = self._rbuf.find('\n') |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
393 while i < 0 and not (0 < limit <= len(self._rbuf)): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
394 new = self._raw_read(self._rbufsize) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
395 if not new: break |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
396 i = new.find('\n') |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
397 if i >= 0: i = i + len(self._rbuf) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
398 self._rbuf = self._rbuf + new |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
399 if i < 0: i = len(self._rbuf) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
400 else: i = i+1 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
401 if 0 <= limit < len(self._rbuf): i = limit |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
402 data, self._rbuf = self._rbuf[:i], self._rbuf[i:] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
403 return data |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
404 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
405 def readlines(self, sizehint = 0): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
406 total = 0 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
407 list = [] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
408 while 1: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
409 line = self.readline() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
410 if not line: break |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
411 list.append(line) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
412 total += len(line) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
413 if sizehint and total >= sizehint: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
414 break |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
415 return list |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
416 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
417 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
418 class HTTPConnection(httplib.HTTPConnection): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
419 # use the modified response class |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
420 response_class = HTTPResponse |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
421 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
422 ######################################################################### |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
423 ##### TEST FUNCTIONS |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
424 ######################################################################### |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
425 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
426 def error_handler(url): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
427 global HANDLE_ERRORS |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
428 orig = HANDLE_ERRORS |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
429 keepalive_handler = HTTPHandler() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
430 opener = urllib2.build_opener(keepalive_handler) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
431 urllib2.install_opener(opener) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
432 pos = {0: 'off', 1: 'on'} |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
433 for i in (0, 1): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
434 print " fancy error handling %s (HANDLE_ERRORS = %i)" % (pos[i], i) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
435 HANDLE_ERRORS = i |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
436 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
437 fo = urllib2.urlopen(url) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
438 foo = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
439 fo.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
440 try: status, reason = fo.status, fo.reason |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
441 except AttributeError: status, reason = None, None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
442 except IOError, e: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
443 print " EXCEPTION: %s" % e |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
444 raise |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
445 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
446 print " status = %s, reason = %s" % (status, reason) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
447 HANDLE_ERRORS = orig |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
448 hosts = keepalive_handler.open_connections() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
449 print "open connections:", hosts |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
450 keepalive_handler.close_all() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
451 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
452 def continuity(url): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
453 import md5 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
454 format = '%25s: %s' |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
455 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
456 # first fetch the file with the normal http handler |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
457 opener = urllib2.build_opener() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
458 urllib2.install_opener(opener) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
459 fo = urllib2.urlopen(url) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
460 foo = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
461 fo.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
462 m = md5.new(foo) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
463 print format % ('normal urllib', m.hexdigest()) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
464 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
465 # now install the keepalive handler and try again |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
466 opener = urllib2.build_opener(HTTPHandler()) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
467 urllib2.install_opener(opener) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
468 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
469 fo = urllib2.urlopen(url) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
470 foo = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
471 fo.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
472 m = md5.new(foo) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
473 print format % ('keepalive read', m.hexdigest()) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
474 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
475 fo = urllib2.urlopen(url) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
476 foo = '' |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
477 while 1: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
478 f = fo.readline() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
479 if f: foo = foo + f |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
480 else: break |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
481 fo.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
482 m = md5.new(foo) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
483 print format % ('keepalive readline', m.hexdigest()) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
484 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
485 def comp(N, url): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
486 print ' making %i connections to:\n %s' % (N, url) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
487 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
488 sys.stdout.write(' first using the normal urllib handlers') |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
489 # first use normal opener |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
490 opener = urllib2.build_opener() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
491 urllib2.install_opener(opener) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
492 t1 = fetch(N, url) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
493 print ' TIME: %.3f s' % t1 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
494 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
495 sys.stdout.write(' now using the keepalive handler ') |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
496 # now install the keepalive handler and try again |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
497 opener = urllib2.build_opener(HTTPHandler()) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
498 urllib2.install_opener(opener) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
499 t2 = fetch(N, url) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
500 print ' TIME: %.3f s' % t2 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
501 print ' improvement factor: %.2f' % (t1/t2, ) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
502 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
503 def fetch(N, url, delay=0): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
504 import time |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
505 lens = [] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
506 starttime = time.time() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
507 for i in range(N): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
508 if delay and i > 0: time.sleep(delay) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
509 fo = urllib2.urlopen(url) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
510 foo = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
511 fo.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
512 lens.append(len(foo)) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
513 diff = time.time() - starttime |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
514 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
515 j = 0 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
516 for i in lens[1:]: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
517 j = j + 1 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
518 if not i == lens[0]: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
519 print "WARNING: inconsistent length on read %i: %i" % (j, i) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
520 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
521 return diff |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
522 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
523 def test_timeout(url): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
524 global DEBUG |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
525 dbbackup = DEBUG |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
526 class FakeLogger: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
527 def debug(self, msg, *args): print msg % args |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
528 info = warning = error = debug |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
529 DEBUG = FakeLogger() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
530 print " fetching the file to establish a connection" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
531 fo = urllib2.urlopen(url) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
532 data1 = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
533 fo.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
534 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
535 i = 20 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
536 print " waiting %i seconds for the server to close the connection" % i |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
537 while i > 0: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
538 sys.stdout.write('\r %2i' % i) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
539 sys.stdout.flush() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
540 time.sleep(1) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
541 i -= 1 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
542 sys.stderr.write('\r') |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
543 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
544 print " fetching the file a second time" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
545 fo = urllib2.urlopen(url) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
546 data2 = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
547 fo.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
548 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
549 if data1 == data2: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
550 print ' data are identical' |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
551 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
552 print ' ERROR: DATA DIFFER' |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
553 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
554 DEBUG = dbbackup |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
555 |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
556 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
557 def test(url, N=10): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
558 print "checking error hander (do this on a non-200)" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
559 try: error_handler(url) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
560 except IOError, e: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
561 print "exiting - exception will prevent further tests" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
562 sys.exit() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
563 print |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
564 print "performing continuity test (making sure stuff isn't corrupted)" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
565 continuity(url) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
566 print |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
567 print "performing speed comparison" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
568 comp(N, url) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
569 print |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
570 print "performing dropped-connection check" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
571 test_timeout(url) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
572 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
573 if __name__ == '__main__': |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
574 import time |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
575 import sys |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
576 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
577 N = int(sys.argv[1]) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
578 url = sys.argv[2] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
579 except: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
580 print "%s <integer> <url>" % sys.argv[0] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
581 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
582 test(url, N) |