Mercurial > hg
annotate mercurial/keepalive.py @ 38162:bdf344aea0ee
extensions: peek command table of disabled extensions without importing
With chg where demandimport disabled, and if disk cache not warm, it took
more than 5 seconds to get "unknown command" error when you typo a command
name. This is horrible UX.
The new implementation is less accurate than the original one as Python
can do anything at import time and cmdtable may be imported from another
module, but I think it's good enough.
Note that the new implementation has to parse .py files, which is slightly
slower than executing .pyc if demandimport is enabled.
author | Yuya Nishihara <yuya@tcha.org> |
---|---|
date | Thu, 03 May 2018 18:38:02 +0900 |
parents | e266e75d77dc |
children | d6d094259d9c |
rev | line source |
---|---|
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
1 # This library is free software; you can redistribute it and/or |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
2 # modify it under the terms of the GNU Lesser General Public |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
3 # License as published by the Free Software Foundation; either |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
4 # version 2.1 of the License, or (at your option) any later version. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
5 # |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
6 # This library is distributed in the hope that it will be useful, |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
7 # but WITHOUT ANY WARRANTY; without even the implied warranty of |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
8 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
9 # Lesser General Public License for more details. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
10 # |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
11 # You should have received a copy of the GNU Lesser General Public |
15782
7de7630053cb
Remove FSF mailing address from GPL headers
Martin Geisler <mg@aragost.com>
parents:
14958
diff
changeset
|
12 # License along with this library; if not, see |
7de7630053cb
Remove FSF mailing address from GPL headers
Martin Geisler <mg@aragost.com>
parents:
14958
diff
changeset
|
13 # <http://www.gnu.org/licenses/>. |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
14 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
15 # This file is part of urlgrabber, a high-level cross-protocol url-grabber |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
16 # Copyright 2002-2004 Michael D. Stenner, Ryan Tomayko |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
17 |
4026
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
18 # Modified by Benoit Boissinot: |
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
19 # - fix for digest auth (inspired from urllib2.py @ Python v2.4) |
6470
ac0bcd951c2c
python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
6001
diff
changeset
|
20 # Modified by Dirkjan Ochtman: |
ac0bcd951c2c
python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
6001
diff
changeset
|
21 # - import md5 function from a local util module |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
22 # Modified by Augie Fackler: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
23 # - add safesend method and use it to prevent broken pipe errors |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
24 # on large POST requests |
4026
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
25 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
26 """An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
27 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
28 >>> import urllib2 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
29 >>> from keepalive import HTTPHandler |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
30 >>> keepalive_handler = HTTPHandler() |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
31 >>> opener = urlreq.buildopener(keepalive_handler) |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
32 >>> urlreq.installopener(opener) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
33 >>> |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
34 >>> fo = urlreq.urlopen('http://www.python.org') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
35 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
36 If a connection to a given host is requested, and all of the existing |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
37 connections are still in use, another connection will be opened. If |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
38 the handler tries to use an existing connection but it fails in some |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
39 way, it will be closed and removed from the pool. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
40 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
41 To remove the handler, simply re-run build_opener with no arguments, and |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
42 install that opener. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
43 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
44 You can explicitly close connections by using the close_connection() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
45 method of the returned file-like object (described below) or you can |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
46 use the handler methods: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
47 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
48 close_connection(host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
49 close_all() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
50 open_connections() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
51 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
52 NOTE: using the close_connection and close_all methods of the handler |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
53 should be done with care when using multiple threads. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
54 * there is nothing that prevents another thread from creating new |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
55 connections immediately after connections are closed |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
56 * no checks are done to prevent in-use connections from being closed |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
57 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
58 >>> keepalive_handler.close_all() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
59 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
60 EXTRA ATTRIBUTES AND METHODS |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
61 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
62 Upon a status of 200, the object returned has a few additional |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
63 attributes and methods, which should not be used if you want to |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
64 remain consistent with the normal urllib2-returned objects: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
65 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
66 close_connection() - close the connection to the host |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
67 readlines() - you know, readlines() |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
17251
diff
changeset
|
68 status - the return status (i.e. 404) |
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
17251
diff
changeset
|
69 reason - english translation of status (i.e. 'File not found') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
70 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
71 If you want the best of both worlds, use this inside an |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
72 AttributeError-catching try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
73 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
74 >>> try: status = fo.status |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
75 >>> except AttributeError: status = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
76 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
77 Unfortunately, these are ONLY there if status == 200, so it's not |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
78 easy to distinguish between non-200 responses. The reason is that |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
79 urllib2 tries to do clever things with error codes 301, 302, 401, |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
80 and 407, and it wraps the object upon return. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
81 """ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
82 |
2444
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
83 # $Id: keepalive.py,v 1.14 2006/04/04 21:00:32 mstenner Exp $ |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
84 |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
85 from __future__ import absolute_import, print_function |
27507
a16489f9132d
keepalive: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
86 |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
87 import errno |
29341
0d83ad967bf8
cleanup: replace uses of util.(md5|sha1|sha256|sha512) with hashlib.\1
Augie Fackler <raf@durin42.com>
parents:
28883
diff
changeset
|
88 import hashlib |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
89 import socket |
27507
a16489f9132d
keepalive: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
90 import sys |
29456
e61d384e3be4
keepalive: switch from thread to threading module
Pulkit Goyal <7895pulkit@gmail.com>
parents:
29455
diff
changeset
|
91 import threading |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
92 |
34308
9bd003052d55
keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32668
diff
changeset
|
93 from .i18n import _ |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
94 from . import ( |
35582
72b91f905065
py3: use node.hex(h.digest()) instead of h.hexdigest()
Pulkit Goyal <7895pulkit@gmail.com>
parents:
35364
diff
changeset
|
95 node, |
34427
a454123f5d94
keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents:
34331
diff
changeset
|
96 pycompat, |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
97 urllibcompat, |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
98 util, |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
99 ) |
37119
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
100 from .utils import ( |
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
101 procutil, |
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
102 ) |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
103 |
29455
0c741fd6158a
py3: conditionalize httplib import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
29341
diff
changeset
|
104 httplib = util.httplib |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
105 urlerr = util.urlerr |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
106 urlreq = util.urlreq |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
107 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
108 DEBUG = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
109 |
14764
a7d5816087a9
classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents:
14494
diff
changeset
|
110 class ConnectionManager(object): |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
111 """ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
112 The connection manager must be able to: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
113 * keep track of all existing |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
114 """ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
115 def __init__(self): |
29456
e61d384e3be4
keepalive: switch from thread to threading module
Pulkit Goyal <7895pulkit@gmail.com>
parents:
29455
diff
changeset
|
116 self._lock = threading.Lock() |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
117 self._hostmap = {} # map hosts to a list of connections |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
118 self._connmap = {} # map connections to host |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
119 self._readymap = {} # map connection to ready state |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
120 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
121 def add(self, host, connection, ready): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
122 self._lock.acquire() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
123 try: |
16686
67964cda8701
cleanup: "not x in y" -> "x not in y"
Brodie Rao <brodie@sf.io>
parents:
15782
diff
changeset
|
124 if host not in self._hostmap: |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
125 self._hostmap[host] = [] |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
126 self._hostmap[host].append(connection) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
127 self._connmap[connection] = host |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
128 self._readymap[connection] = ready |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
129 finally: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
130 self._lock.release() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
131 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
132 def remove(self, connection): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
133 self._lock.acquire() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
134 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
135 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
136 host = self._connmap[connection] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
137 except KeyError: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
138 pass |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
139 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
140 del self._connmap[connection] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
141 del self._readymap[connection] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
142 self._hostmap[host].remove(connection) |
34435
5326e4ef1dab
style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents:
34427
diff
changeset
|
143 if not self._hostmap[host]: |
5326e4ef1dab
style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents:
34427
diff
changeset
|
144 del self._hostmap[host] |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
145 finally: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
146 self._lock.release() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
147 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
148 def set_ready(self, connection, ready): |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
149 try: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
150 self._readymap[connection] = ready |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
151 except KeyError: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
152 pass |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
153 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
154 def get_ready_conn(self, host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
155 conn = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
156 self._lock.acquire() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
157 try: |
5915
d0576d065993
Prefer i in d over d.has_key(i)
Christian Ebert <blacktrash@gmx.net>
parents:
4026
diff
changeset
|
158 if host in self._hostmap: |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
159 for c in self._hostmap[host]: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
160 if self._readymap[c]: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
161 self._readymap[c] = 0 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
162 conn = c |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
163 break |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
164 finally: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
165 self._lock.release() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
166 return conn |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
167 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
168 def get_all(self, host=None): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
169 if host: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
170 return list(self._hostmap.get(host, [])) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
171 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
172 return dict(self._hostmap) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
173 |
14764
a7d5816087a9
classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents:
14494
diff
changeset
|
174 class KeepAliveHandler(object): |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
175 def __init__(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
176 self._cm = ConnectionManager() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
177 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
178 #### Connection Management |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
179 def open_connections(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
180 """return a list of connected hosts and the number of connections |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
181 to each. [('foo.com:80', 2), ('bar.org', 1)]""" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
182 return [(host, len(li)) for (host, li) in self._cm.get_all().items()] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
183 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
184 def close_connection(self, host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
185 """close connection(s) to <host> |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
186 host is the host:port spec, as in 'www.cnn.com:8080' as passed in. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
187 no error occurs if there is no connection to that host.""" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
188 for h in self._cm.get_all(host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
189 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
190 h.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
191 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
192 def close_all(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
193 """close all open connections""" |
7622
4dd7b28003d2
use dict.iteritems() rather than dict.items()
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
6470
diff
changeset
|
194 for host, conns in self._cm.get_all().iteritems(): |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
195 for h in conns: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
196 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
197 h.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
198 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
199 def _request_closed(self, request, host, connection): |
17251
98166640b356
help: fix some instances of 'the the'
Mads Kiilerich <mads@kiilerich.com>
parents:
16705
diff
changeset
|
200 """tells us that this request is now closed and that the |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
201 connection is ready for another request""" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
202 self._cm.set_ready(connection, 1) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
203 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
204 def _remove_connection(self, host, connection, close=0): |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
205 if close: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
206 connection.close() |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
207 self._cm.remove(connection) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
208 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
209 #### Transaction Execution |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
210 def http_open(self, req): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
211 return self.do_open(HTTPConnection, req) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
212 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
213 def do_open(self, http_class, req): |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
214 host = urllibcompat.gethost(req) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
215 if not host: |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
216 raise urlerr.urlerror('no host given') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
217 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
218 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
219 h = self._cm.get_ready_conn(host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
220 while h: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
221 r = self._reuse_connection(h, req, host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
222 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
223 # if this response is non-None, then it worked and we're |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
224 # done. Break out, skipping the else block. |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
225 if r: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
226 break |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
227 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
228 # connection is bad - possibly closed by server |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
229 # discard it and ask for the next free connection |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
230 h.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
231 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
232 h = self._cm.get_ready_conn(host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
233 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
234 # no (working) free connections were found. Create a new one. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
235 h = http_class(host) |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
236 if DEBUG: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
237 DEBUG.info("creating new connection to %s (%d)", |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
238 host, id(h)) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
239 self._cm.add(host, h, 0) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
240 self._start_transaction(h, req) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
241 r = h.getresponse() |
34308
9bd003052d55
keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32668
diff
changeset
|
242 # The string form of BadStatusLine is the status line. Add some context |
9bd003052d55
keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32668
diff
changeset
|
243 # to make the error message slightly more useful. |
9bd003052d55
keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32668
diff
changeset
|
244 except httplib.BadStatusLine as err: |
34427
a454123f5d94
keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents:
34331
diff
changeset
|
245 raise urlerr.urlerror( |
a454123f5d94
keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents:
34331
diff
changeset
|
246 _('bad HTTP status line: %s') % pycompat.sysbytes(err.line)) |
25660
328739ea70c3
global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents:
22958
diff
changeset
|
247 except (socket.error, httplib.HTTPException) as err: |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
248 raise urlerr.urlerror(err) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
249 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
250 # if not a persistent connection, don't try to reuse it |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
251 if r.will_close: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
252 self._cm.remove(h) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
253 |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
254 if DEBUG: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
255 DEBUG.info("STATUS: %s, %s", r.status, r.reason) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
256 r._handler = self |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
257 r._host = host |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
258 r._url = req.get_full_url() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
259 r._connection = h |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
260 r.code = r.status |
2444
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
261 r.headers = r.msg |
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
262 r.msg = r.reason |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
263 |
30487
88a448a12ae8
keepalive: discard legacy Python support for error handling
Augie Fackler <augie@google.com>
parents:
30473
diff
changeset
|
264 return r |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
265 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
266 def _reuse_connection(self, h, req, host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
267 """start the transaction with a re-used connection |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
268 return a response object (r) upon success or None on failure. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
269 This DOES not close or remove bad connections in cases where |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
270 it returns. However, if an unexpected exception occurs, it |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
271 will close and remove the connection before re-raising. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
272 """ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
273 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
274 self._start_transaction(h, req) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
275 r = h.getresponse() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
276 # note: just because we got something back doesn't mean it |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
277 # worked. We'll check the version below, too. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
278 except (socket.error, httplib.HTTPException): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
279 r = None |
16705
c2d9ef43ff6c
check-code: ignore naked excepts with a "re-raise" comment
Brodie Rao <brodie@sf.io>
parents:
16688
diff
changeset
|
280 except: # re-raises |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
281 # adding this block just in case we've missed |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
282 # something we will still raise the exception, but |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
283 # lets try and close the connection and remove it |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
284 # first. We previously got into a nasty loop |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
285 # where an exception was uncaught, and so the |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
286 # connection stayed open. On the next try, the |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
17251
diff
changeset
|
287 # same exception was raised, etc. The trade-off is |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
288 # that it's now possible this call will raise |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
289 # a DIFFERENT exception |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
290 if DEBUG: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
291 DEBUG.error("unexpected exception - closing " |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
292 "connection to %s (%d)", host, id(h)) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
293 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
294 h.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
295 raise |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
296 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
297 if r is None or r.version == 9: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
298 # httplib falls back to assuming HTTP 0.9 if it gets a |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
299 # bad header back. This is most likely to happen if |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
300 # the socket has been closed by the server since we |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
301 # last used the connection. |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
302 if DEBUG: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
303 DEBUG.info("failed to re-use connection to %s (%d)", |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
304 host, id(h)) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
305 r = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
306 else: |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
307 if DEBUG: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
308 DEBUG.info("re-using connection to %s (%d)", host, id(h)) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
309 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
310 return r |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
311 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
312 def _start_transaction(self, h, req): |
8233
655c435efe92
keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents:
8146
diff
changeset
|
313 # What follows mostly reimplements HTTPConnection.request() |
31999
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
314 # except it adds self.parent.addheaders in the mix and sends headers |
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
315 # in a deterministic order (to make testing easier). |
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
316 headers = util.sortdict(self.parent.addheaders) |
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
317 headers.update(sorted(req.headers.items())) |
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
318 headers.update(sorted(req.unredirected_hdrs.items())) |
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
319 headers = util.sortdict((n.lower(), v) for n, v in headers.items()) |
8233
655c435efe92
keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents:
8146
diff
changeset
|
320 skipheaders = {} |
37665
83250442dc81
py3: use str variables to check keys in request header
Pulkit Goyal <7895pulkit@gmail.com>
parents:
37587
diff
changeset
|
321 for n in (r'host', r'accept-encoding'): |
8233
655c435efe92
keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents:
8146
diff
changeset
|
322 if n in headers: |
37665
83250442dc81
py3: use str variables to check keys in request header
Pulkit Goyal <7895pulkit@gmail.com>
parents:
37587
diff
changeset
|
323 skipheaders[r'skip_' + n.replace(r'-', r'_')] = 1 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
324 try: |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
325 if urllibcompat.hasdata(req): |
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
326 data = urllibcompat.getdata(req) |
30922
1beeb5185930
keepalive: honor urllib2 style get_method overrides
John Mulligan <phlogistonjohn@asynchrono.us>
parents:
30688
diff
changeset
|
327 h.putrequest( |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
328 req.get_method(), urllibcompat.getselector(req), |
37688
e266e75d77dc
keepalive: add ** overlooked in 83250442dc81
Augie Fackler <augie@google.com>
parents:
37665
diff
changeset
|
329 **skipheaders) |
36295
19a04ca90413
keepalive: headers are native strings, mark them as such
Augie Fackler <augie@google.com>
parents:
35582
diff
changeset
|
330 if r'content-type' not in headers: |
19a04ca90413
keepalive: headers are native strings, mark them as such
Augie Fackler <augie@google.com>
parents:
35582
diff
changeset
|
331 h.putheader(r'Content-type', |
19a04ca90413
keepalive: headers are native strings, mark them as such
Augie Fackler <augie@google.com>
parents:
35582
diff
changeset
|
332 r'application/x-www-form-urlencoded') |
19a04ca90413
keepalive: headers are native strings, mark them as such
Augie Fackler <augie@google.com>
parents:
35582
diff
changeset
|
333 if r'content-length' not in headers: |
19a04ca90413
keepalive: headers are native strings, mark them as such
Augie Fackler <augie@google.com>
parents:
35582
diff
changeset
|
334 h.putheader(r'Content-length', r'%d' % len(data)) |
8146
4f13ed6ee544
keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents:
7875
diff
changeset
|
335 else: |
30922
1beeb5185930
keepalive: honor urllib2 style get_method overrides
John Mulligan <phlogistonjohn@asynchrono.us>
parents:
30688
diff
changeset
|
336 h.putrequest( |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
337 req.get_method(), urllibcompat.getselector(req), |
37688
e266e75d77dc
keepalive: add ** overlooked in 83250442dc81
Augie Fackler <augie@google.com>
parents:
37665
diff
changeset
|
338 **skipheaders) |
28278
b1b22185c764
keepalive: remove useless parentheses around exception type
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27616
diff
changeset
|
339 except socket.error as err: |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
340 raise urlerr.urlerror(err) |
8233
655c435efe92
keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents:
8146
diff
changeset
|
341 for k, v in headers.items(): |
8146
4f13ed6ee544
keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents:
7875
diff
changeset
|
342 h.putheader(k, v) |
4f13ed6ee544
keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents:
7875
diff
changeset
|
343 h.endheaders() |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
344 if urllibcompat.hasdata(req): |
8146
4f13ed6ee544
keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents:
7875
diff
changeset
|
345 h.send(data) |
4f13ed6ee544
keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents:
7875
diff
changeset
|
346 |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
347 class HTTPHandler(KeepAliveHandler, urlreq.httphandler): |
5983
6f1fcbc58efa
httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
4026
diff
changeset
|
348 pass |
6f1fcbc58efa
httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
4026
diff
changeset
|
349 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
350 class HTTPResponse(httplib.HTTPResponse): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
351 # we need to subclass HTTPResponse in order to |
37297
97eedbd5a56c
keepalive: implement readinto()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37119
diff
changeset
|
352 # 1) add readline(), readlines(), and readinto() methods |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
353 # 2) add close_connection() methods |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
354 # 3) add info() and geturl() methods |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
355 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
356 # in order to add readline(), read must be modified to deal with a |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
357 # buffer. example: readline must read a buffer and then spit back |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
358 # one line at a time. The only real alternative is to read one |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
359 # BYTE at a time (ick). Once something has been read, it can't be |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
360 # put back (ok, maybe it can, but that's even uglier than this), |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
361 # so if you THEN do a normal read, you must first take stuff from |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
362 # the buffer. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
363 |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
17251
diff
changeset
|
364 # the read method wraps the original to accommodate buffering, |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
365 # although read() never adds to the buffer. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
366 # Both readline and readlines have been stolen with almost no |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
367 # modification from socket.py |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
368 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
369 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
370 def __init__(self, sock, debuglevel=0, strict=0, method=None): |
34427
a454123f5d94
keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents:
34331
diff
changeset
|
371 extrakw = {} |
a454123f5d94
keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents:
34331
diff
changeset
|
372 if not pycompat.ispy3: |
35364
03112a2c9c83
py3: handle keyword arguments correctly in keepalive.py
Pulkit Goyal <7895pulkit@gmail.com>
parents:
34466
diff
changeset
|
373 extrakw[r'strict'] = True |
03112a2c9c83
py3: handle keyword arguments correctly in keepalive.py
Pulkit Goyal <7895pulkit@gmail.com>
parents:
34466
diff
changeset
|
374 extrakw[r'buffering'] = True |
32667
2806c7bbcb5e
keepalive: pass the correct arguments to HTTPResponse
Kyle Lippincott <spectral@google.com>
parents:
31999
diff
changeset
|
375 httplib.HTTPResponse.__init__(self, sock, debuglevel=debuglevel, |
34427
a454123f5d94
keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents:
34331
diff
changeset
|
376 method=method, **extrakw) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
377 self.fileno = sock.fileno |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
378 self.code = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
379 self._rbuf = '' |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
380 self._rbufsize = 8096 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
381 self._handler = None # inserted by the handler later |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
382 self._host = None # (same) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
383 self._url = None # (same) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
384 self._connection = None # (same) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
385 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
386 _raw_read = httplib.HTTPResponse.read |
37587
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
387 _raw_readinto = getattr(httplib.HTTPResponse, 'readinto', None) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
388 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
389 def close(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
390 if self.fp: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
391 self.fp.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
392 self.fp = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
393 if self._handler: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
394 self._handler._request_closed(self, self._host, |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
395 self._connection) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
396 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
397 def close_connection(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
398 self._handler._remove_connection(self._host, self._connection, close=1) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
399 self.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
400 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
401 def info(self): |
2444
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
402 return self.headers |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
403 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
404 def geturl(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
405 return self._url |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
406 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
407 def read(self, amt=None): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
408 # the _rbuf test is only in this first if for speed. It's not |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
409 # logically necessary |
34331
531332502568
style: always use `x is not None` instead of `not x is None`
Alex Gaynor <agaynor@mozilla.com>
parents:
34308
diff
changeset
|
410 if self._rbuf and amt is not None: |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
411 L = len(self._rbuf) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
412 if amt > L: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
413 amt -= L |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
414 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
415 s = self._rbuf[:amt] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
416 self._rbuf = self._rbuf[amt:] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
417 return s |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
418 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
419 s = self._rbuf + self._raw_read(amt) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
420 self._rbuf = '' |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
421 return s |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
422 |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
423 # stolen from Python SVN #68532 to fix issue1088 |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
424 def _read_chunked(self, amt): |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
425 chunk_left = self.chunk_left |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
426 parts = [] |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
427 |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
428 while True: |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
429 if chunk_left is None: |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
430 line = self.fp.readline() |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
431 i = line.find(';') |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
432 if i >= 0: |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
433 line = line[:i] # strip chunk-extensions |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
434 try: |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
435 chunk_left = int(line, 16) |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
436 except ValueError: |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
17251
diff
changeset
|
437 # close the connection as protocol synchronization is |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
438 # probably lost |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
439 self.close() |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
440 raise httplib.IncompleteRead(''.join(parts)) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
441 if chunk_left == 0: |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
442 break |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
443 if amt is None: |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
444 parts.append(self._safe_read(chunk_left)) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
445 elif amt < chunk_left: |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
446 parts.append(self._safe_read(amt)) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
447 self.chunk_left = chunk_left - amt |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
448 return ''.join(parts) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
449 elif amt == chunk_left: |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
450 parts.append(self._safe_read(amt)) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
451 self._safe_read(2) # toss the CRLF at the end of the chunk |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
452 self.chunk_left = None |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
453 return ''.join(parts) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
454 else: |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
455 parts.append(self._safe_read(chunk_left)) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
456 amt -= chunk_left |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
457 |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
458 # we read the whole chunk, get another |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
459 self._safe_read(2) # toss the CRLF at the end of the chunk |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
460 chunk_left = None |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
461 |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
462 # read and discard trailer up to the CRLF terminator |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
463 ### note: we shouldn't have any trailers! |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
464 while True: |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
465 line = self.fp.readline() |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
466 if not line: |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
467 # a vanishingly small number of sites EOF without |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
468 # sending the trailer |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
469 break |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
470 if line == '\r\n': |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
471 break |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
472 |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
473 # we read everything; close the "file" |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
474 self.close() |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
475 |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
476 return ''.join(parts) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
477 |
30687
5d06f6b73a57
keepalive: remove limit argument from readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30686
diff
changeset
|
478 def readline(self): |
30688
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
479 # Fast path for a line is already available in read buffer. |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
480 i = self._rbuf.find('\n') |
30688
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
481 if i >= 0: |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
482 i += 1 |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
483 line = self._rbuf[:i] |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
484 self._rbuf = self._rbuf[i:] |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
485 return line |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
486 |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
487 # No newline in local buffer. Read until we find one. |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
488 chunks = [self._rbuf] |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
489 i = -1 |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
490 readsize = self._rbufsize |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
491 while True: |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
492 new = self._raw_read(readsize) |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
493 if not new: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
494 break |
30688
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
495 |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
496 chunks.append(new) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
497 i = new.find('\n') |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
498 if i >= 0: |
30688
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
499 break |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
500 |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
501 # We either have exhausted the stream or have a newline in chunks[-1]. |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
502 |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
503 # EOF |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
504 if i == -1: |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
505 self._rbuf = '' |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
506 return ''.join(chunks) |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
507 |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
508 i += 1 |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
509 self._rbuf = chunks[-1][i:] |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
510 chunks[-1] = chunks[-1][:i] |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
511 return ''.join(chunks) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
512 |
19872
681f7b9213a4
check-code: check for spaces around = for named parameters
Mads Kiilerich <madski@unity3d.com>
parents:
17700
diff
changeset
|
513 def readlines(self, sizehint=0): |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
514 total = 0 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
515 list = [] |
14494
1ffeeb91c55d
check-code: flag 0/1 used as constant Boolean expression
Martin Geisler <mg@lazybytes.net>
parents:
10394
diff
changeset
|
516 while True: |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
517 line = self.readline() |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
518 if not line: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
519 break |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
520 list.append(line) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
521 total += len(line) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
522 if sizehint and total >= sizehint: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
523 break |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
524 return list |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
525 |
37297
97eedbd5a56c
keepalive: implement readinto()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37119
diff
changeset
|
526 def readinto(self, dest): |
37587
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
527 if self._raw_readinto is None: |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
528 res = self.read(len(dest)) |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
529 if not res: |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
530 return 0 |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
531 dest[0:len(res)] = res |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
532 return len(res) |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
533 total = len(dest) |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
534 have = len(self._rbuf) |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
535 if have >= total: |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
536 dest[0:total] = self._rbuf[:total] |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
537 self._rbuf = self._rbuf[total:] |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
538 return total |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
539 mv = memoryview(dest) |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
540 got = self._raw_readinto(mv[have:total]) |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
541 dest[0:have] = self._rbuf |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
542 got += len(self._rbuf) |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
543 self._rbuf = '' |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
544 return got |
37297
97eedbd5a56c
keepalive: implement readinto()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37119
diff
changeset
|
545 |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
546 def safesend(self, str): |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
547 """Send `str' to the server. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
548 |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
549 Shamelessly ripped off from httplib to patch a bad behavior. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
550 """ |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
551 # _broken_pipe_resp is an attribute we set in this function |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
552 # if the socket is closed while we're sending data but |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
553 # the server sent us a response before hanging up. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
554 # In that case, we want to pretend to send the rest of the |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
555 # outgoing data, and then let the user use getresponse() |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
556 # (which we wrap) to get this last response before |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
557 # opening a new socket. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
558 if getattr(self, '_broken_pipe_resp', None) is not None: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
559 return |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
560 |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
561 if self.sock is None: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
562 if self.auto_open: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
563 self.connect() |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
564 else: |
16687
e34106fa0dc3
cleanup: "raise SomeException()" -> "raise SomeException"
Brodie Rao <brodie@sf.io>
parents:
16686
diff
changeset
|
565 raise httplib.NotConnected |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
566 |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
567 # send the data to the server. if we get a broken pipe, then close |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
568 # the socket. we want to reconnect when somebody tries to send again. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
569 # |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
570 # NOTE: we DO propagate the error, though, because we cannot simply |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
571 # ignore the error... the caller will know if they can retry. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
572 if self.debuglevel > 0: |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
573 print("send:", repr(str)) |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
574 try: |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
575 blocksize = 8192 |
14958
fd246aefedd3
keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents:
14764
diff
changeset
|
576 read = getattr(str, 'read', None) |
fd246aefedd3
keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents:
14764
diff
changeset
|
577 if read is not None: |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
578 if self.debuglevel > 0: |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
579 print("sending a read()able") |
14958
fd246aefedd3
keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents:
14764
diff
changeset
|
580 data = read(blocksize) |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
581 while data: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
582 self.sock.sendall(data) |
14958
fd246aefedd3
keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents:
14764
diff
changeset
|
583 data = read(blocksize) |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
584 else: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
585 self.sock.sendall(str) |
25660
328739ea70c3
global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents:
22958
diff
changeset
|
586 except socket.error as v: |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
587 reraise = True |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
588 if v[0] == errno.EPIPE: # Broken pipe |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
589 if self._HTTPConnection__state == httplib._CS_REQ_SENT: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
590 self._broken_pipe_resp = None |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
591 self._broken_pipe_resp = self.getresponse() |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
592 reraise = False |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
593 self.close() |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
594 if reraise: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
595 raise |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
596 |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
597 def wrapgetresponse(cls): |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
598 """Wraps getresponse in cls with a broken-pipe sane version. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
599 """ |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
600 def safegetresponse(self): |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
601 # In safesend() we might set the _broken_pipe_resp |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
602 # attribute, in which case the socket has already |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
603 # been closed and we just need to give them the response |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
604 # back. Otherwise, we use the normal response path. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
605 r = getattr(self, '_broken_pipe_resp', None) |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
606 if r is not None: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
607 return r |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
608 return cls.getresponse(self) |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
609 safegetresponse.__doc__ = cls.getresponse.__doc__ |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
610 return safegetresponse |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
611 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
612 class HTTPConnection(httplib.HTTPConnection): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
613 # use the modified response class |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
614 response_class = HTTPResponse |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
615 send = safesend |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
616 getresponse = wrapgetresponse(httplib.HTTPConnection) |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
617 |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
618 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
619 ######################################################################### |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
620 ##### TEST FUNCTIONS |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
621 ######################################################################### |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
622 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
623 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
624 def continuity(url): |
29341
0d83ad967bf8
cleanup: replace uses of util.(md5|sha1|sha256|sha512) with hashlib.\1
Augie Fackler <raf@durin42.com>
parents:
28883
diff
changeset
|
625 md5 = hashlib.md5 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
626 format = '%25s: %s' |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
627 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
628 # first fetch the file with the normal http handler |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
629 opener = urlreq.buildopener() |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
630 urlreq.installopener(opener) |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
631 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
632 foo = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
633 fo.close() |
22505
232d437af120
keepalive: fix how md5 is used
Mike Hommey <mh@glandium.org>
parents:
19872
diff
changeset
|
634 m = md5(foo) |
35582
72b91f905065
py3: use node.hex(h.digest()) instead of h.hexdigest()
Pulkit Goyal <7895pulkit@gmail.com>
parents:
35364
diff
changeset
|
635 print(format % ('normal urllib', node.hex(m.digest()))) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
636 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
637 # now install the keepalive handler and try again |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
638 opener = urlreq.buildopener(HTTPHandler()) |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
639 urlreq.installopener(opener) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
640 |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
641 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
642 foo = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
643 fo.close() |
22505
232d437af120
keepalive: fix how md5 is used
Mike Hommey <mh@glandium.org>
parents:
19872
diff
changeset
|
644 m = md5(foo) |
35582
72b91f905065
py3: use node.hex(h.digest()) instead of h.hexdigest()
Pulkit Goyal <7895pulkit@gmail.com>
parents:
35364
diff
changeset
|
645 print(format % ('keepalive read', node.hex(m.digest()))) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
646 |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
647 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
648 foo = '' |
14494
1ffeeb91c55d
check-code: flag 0/1 used as constant Boolean expression
Martin Geisler <mg@lazybytes.net>
parents:
10394
diff
changeset
|
649 while True: |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
650 f = fo.readline() |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
651 if f: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
652 foo = foo + f |
34435
5326e4ef1dab
style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents:
34427
diff
changeset
|
653 else: |
5326e4ef1dab
style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents:
34427
diff
changeset
|
654 break |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
655 fo.close() |
22505
232d437af120
keepalive: fix how md5 is used
Mike Hommey <mh@glandium.org>
parents:
19872
diff
changeset
|
656 m = md5(foo) |
35582
72b91f905065
py3: use node.hex(h.digest()) instead of h.hexdigest()
Pulkit Goyal <7895pulkit@gmail.com>
parents:
35364
diff
changeset
|
657 print(format % ('keepalive readline', node.hex(m.digest()))) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
658 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
659 def comp(N, url): |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
660 print(' making %i connections to:\n %s' % (N, url)) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
661 |
37119
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
662 procutil.stdout.write(' first using the normal urllib handlers') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
663 # first use normal opener |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
664 opener = urlreq.buildopener() |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
665 urlreq.installopener(opener) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
666 t1 = fetch(N, url) |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
667 print(' TIME: %.3f s' % t1) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
668 |
37119
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
669 procutil.stdout.write(' now using the keepalive handler ') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
670 # now install the keepalive handler and try again |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
671 opener = urlreq.buildopener(HTTPHandler()) |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
672 urlreq.installopener(opener) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
673 t2 = fetch(N, url) |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
674 print(' TIME: %.3f s' % t2) |
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
675 print(' improvement factor: %.2f' % (t1 / t2)) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
676 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
677 def fetch(N, url, delay=0): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
678 import time |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
679 lens = [] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
680 starttime = time.time() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
681 for i in range(N): |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
682 if delay and i > 0: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
683 time.sleep(delay) |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
684 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
685 foo = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
686 fo.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
687 lens.append(len(foo)) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
688 diff = time.time() - starttime |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
689 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
690 j = 0 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
691 for i in lens[1:]: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
692 j = j + 1 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
693 if not i == lens[0]: |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
694 print("WARNING: inconsistent length on read %i: %i" % (j, i)) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
695 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
696 return diff |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
697 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
698 def test_timeout(url): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
699 global DEBUG |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
700 dbbackup = DEBUG |
14764
a7d5816087a9
classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents:
14494
diff
changeset
|
701 class FakeLogger(object): |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
702 def debug(self, msg, *args): |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
703 print(msg % args) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
704 info = warning = error = debug |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
705 DEBUG = FakeLogger() |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
706 print(" fetching the file to establish a connection") |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
707 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
708 data1 = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
709 fo.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
710 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
711 i = 20 |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
712 print(" waiting %i seconds for the server to close the connection" % i) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
713 while i > 0: |
37119
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
714 procutil.stdout.write('\r %2i' % i) |
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
715 procutil.stdout.flush() |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
716 time.sleep(1) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
717 i -= 1 |
37119
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
718 procutil.stderr.write('\r') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
719 |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
720 print(" fetching the file a second time") |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
721 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
722 data2 = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
723 fo.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
724 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
725 if data1 == data2: |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
726 print(' data are identical') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
727 else: |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
728 print(' ERROR: DATA DIFFER') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
729 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
730 DEBUG = dbbackup |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
731 |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
732 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
733 def test(url, N=10): |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
734 print("performing continuity test (making sure stuff isn't corrupted)") |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
735 continuity(url) |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
736 print('') |
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
737 print("performing speed comparison") |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
738 comp(N, url) |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
739 print('') |
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
740 print("performing dropped-connection check") |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
741 test_timeout(url) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
742 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
743 if __name__ == '__main__': |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
744 import time |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
745 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
746 N = int(sys.argv[1]) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
747 url = sys.argv[2] |
16688
cfb6682961b8
cleanup: replace naked excepts with more specific ones
Brodie Rao <brodie@sf.io>
parents:
16687
diff
changeset
|
748 except (IndexError, ValueError): |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
749 print("%s <integer> <url>" % sys.argv[0]) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
750 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
751 test(url, N) |