Mercurial > hg
annotate mercurial/keepalive.py @ 40324:6637b079ae45
lfs: autoload the extension when cloning from repo with lfs enabled
This is based on a patch by Gregory Szorc. I made small adjustments to
clean up the messaging when the server has the extension enabled, but the
client has it disabled (to prevent autoloading). Additionally, I added
a second server capability to distinguish between the server having the
extension enabled, and the server having LFS commits. This helps prevent
unnecessary requirement propagation- the client shouldn't add a requirement
that the server doesn't have, just because the server had the extension
loaded. The TODO I had about advertising a capability when the server can
natively serve up blobs isn't relevant anymore (we've had 2 releases that
support this), so I dropped it.
Currently, we lazily add the "lfs" requirement to a repo when we first
encounter LFS data. Due to a pretxnchangegroup hook that looks for LFS
data, this can happen at the end of clone.
Now that we have more control over how repositories are created, we can
do better.
This commit adds a repo creation option to add the "lfs" requirement.
hg.clone() sets this creation option if the remote peer is advertising
lfs usage (as opposed to just support needed to push).
So, what this change effectively does is have cloned repos
automatically inherit the "lfs" requirement.
Differential Revision: https://phab.mercurial-scm.org/D5130
author | Matt Harbison <matt_harbison@yahoo.com> |
---|---|
date | Thu, 20 Sep 2018 17:27:01 -0700 |
parents | 6509fcec830c |
children | 41506e3b04ee |
rev | line source |
---|---|
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
1 # This library is free software; you can redistribute it and/or |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
2 # modify it under the terms of the GNU Lesser General Public |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
3 # License as published by the Free Software Foundation; either |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
4 # version 2.1 of the License, or (at your option) any later version. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
5 # |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
6 # This library is distributed in the hope that it will be useful, |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
7 # but WITHOUT ANY WARRANTY; without even the implied warranty of |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
8 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
9 # Lesser General Public License for more details. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
10 # |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
11 # You should have received a copy of the GNU Lesser General Public |
15782
7de7630053cb
Remove FSF mailing address from GPL headers
Martin Geisler <mg@aragost.com>
parents:
14958
diff
changeset
|
12 # License along with this library; if not, see |
7de7630053cb
Remove FSF mailing address from GPL headers
Martin Geisler <mg@aragost.com>
parents:
14958
diff
changeset
|
13 # <http://www.gnu.org/licenses/>. |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
14 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
15 # This file is part of urlgrabber, a high-level cross-protocol url-grabber |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
16 # Copyright 2002-2004 Michael D. Stenner, Ryan Tomayko |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
17 |
4026
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
18 # Modified by Benoit Boissinot: |
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
19 # - fix for digest auth (inspired from urllib2.py @ Python v2.4) |
6470
ac0bcd951c2c
python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
6001
diff
changeset
|
20 # Modified by Dirkjan Ochtman: |
ac0bcd951c2c
python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
6001
diff
changeset
|
21 # - import md5 function from a local util module |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
22 # Modified by Augie Fackler: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
23 # - add safesend method and use it to prevent broken pipe errors |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
24 # on large POST requests |
4026
8520a773a141
fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents:
2600
diff
changeset
|
25 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
26 """An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
27 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
28 >>> import urllib2 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
29 >>> from keepalive import HTTPHandler |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
30 >>> keepalive_handler = HTTPHandler() |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
31 >>> opener = urlreq.buildopener(keepalive_handler) |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
32 >>> urlreq.installopener(opener) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
33 >>> |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
34 >>> fo = urlreq.urlopen('http://www.python.org') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
35 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
36 If a connection to a given host is requested, and all of the existing |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
37 connections are still in use, another connection will be opened. If |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
38 the handler tries to use an existing connection but it fails in some |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
39 way, it will be closed and removed from the pool. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
40 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
41 To remove the handler, simply re-run build_opener with no arguments, and |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
42 install that opener. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
43 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
44 You can explicitly close connections by using the close_connection() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
45 method of the returned file-like object (described below) or you can |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
46 use the handler methods: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
47 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
48 close_connection(host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
49 close_all() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
50 open_connections() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
51 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
52 NOTE: using the close_connection and close_all methods of the handler |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
53 should be done with care when using multiple threads. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
54 * there is nothing that prevents another thread from creating new |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
55 connections immediately after connections are closed |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
56 * no checks are done to prevent in-use connections from being closed |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
57 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
58 >>> keepalive_handler.close_all() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
59 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
60 EXTRA ATTRIBUTES AND METHODS |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
61 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
62 Upon a status of 200, the object returned has a few additional |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
63 attributes and methods, which should not be used if you want to |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
64 remain consistent with the normal urllib2-returned objects: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
65 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
66 close_connection() - close the connection to the host |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
67 readlines() - you know, readlines() |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
17251
diff
changeset
|
68 status - the return status (i.e. 404) |
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
17251
diff
changeset
|
69 reason - english translation of status (i.e. 'File not found') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
70 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
71 If you want the best of both worlds, use this inside an |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
72 AttributeError-catching try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
73 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
74 >>> try: status = fo.status |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
75 >>> except AttributeError: status = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
76 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
77 Unfortunately, these are ONLY there if status == 200, so it's not |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
78 easy to distinguish between non-200 responses. The reason is that |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
79 urllib2 tries to do clever things with error codes 301, 302, 401, |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
80 and 407, and it wraps the object upon return. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
81 """ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
82 |
2444
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
83 # $Id: keepalive.py,v 1.14 2006/04/04 21:00:32 mstenner Exp $ |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
84 |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
85 from __future__ import absolute_import, print_function |
27507
a16489f9132d
keepalive: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
86 |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
87 import errno |
29341
0d83ad967bf8
cleanup: replace uses of util.(md5|sha1|sha256|sha512) with hashlib.\1
Augie Fackler <raf@durin42.com>
parents:
28883
diff
changeset
|
88 import hashlib |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
89 import socket |
27507
a16489f9132d
keepalive: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
90 import sys |
29456
e61d384e3be4
keepalive: switch from thread to threading module
Pulkit Goyal <7895pulkit@gmail.com>
parents:
29455
diff
changeset
|
91 import threading |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
92 |
34308
9bd003052d55
keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32668
diff
changeset
|
93 from .i18n import _ |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
94 from . import ( |
35582
72b91f905065
py3: use node.hex(h.digest()) instead of h.hexdigest()
Pulkit Goyal <7895pulkit@gmail.com>
parents:
35364
diff
changeset
|
95 node, |
34427
a454123f5d94
keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents:
34331
diff
changeset
|
96 pycompat, |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
97 urllibcompat, |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
98 util, |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
99 ) |
37119
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
100 from .utils import ( |
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
101 procutil, |
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
102 ) |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
103 |
29455
0c741fd6158a
py3: conditionalize httplib import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
29341
diff
changeset
|
104 httplib = util.httplib |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
105 urlerr = util.urlerr |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
106 urlreq = util.urlreq |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
107 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
108 DEBUG = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
109 |
14764
a7d5816087a9
classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents:
14494
diff
changeset
|
110 class ConnectionManager(object): |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
111 """ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
112 The connection manager must be able to: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
113 * keep track of all existing |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
114 """ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
115 def __init__(self): |
29456
e61d384e3be4
keepalive: switch from thread to threading module
Pulkit Goyal <7895pulkit@gmail.com>
parents:
29455
diff
changeset
|
116 self._lock = threading.Lock() |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
117 self._hostmap = {} # map hosts to a list of connections |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
118 self._connmap = {} # map connections to host |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
119 self._readymap = {} # map connection to ready state |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
120 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
121 def add(self, host, connection, ready): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
122 self._lock.acquire() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
123 try: |
16686
67964cda8701
cleanup: "not x in y" -> "x not in y"
Brodie Rao <brodie@sf.io>
parents:
15782
diff
changeset
|
124 if host not in self._hostmap: |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
125 self._hostmap[host] = [] |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
126 self._hostmap[host].append(connection) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
127 self._connmap[connection] = host |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
128 self._readymap[connection] = ready |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
129 finally: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
130 self._lock.release() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
131 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
132 def remove(self, connection): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
133 self._lock.acquire() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
134 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
135 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
136 host = self._connmap[connection] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
137 except KeyError: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
138 pass |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
139 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
140 del self._connmap[connection] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
141 del self._readymap[connection] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
142 self._hostmap[host].remove(connection) |
34435
5326e4ef1dab
style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents:
34427
diff
changeset
|
143 if not self._hostmap[host]: |
5326e4ef1dab
style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents:
34427
diff
changeset
|
144 del self._hostmap[host] |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
145 finally: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
146 self._lock.release() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
147 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
148 def set_ready(self, connection, ready): |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
149 try: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
150 self._readymap[connection] = ready |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
151 except KeyError: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
152 pass |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
153 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
154 def get_ready_conn(self, host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
155 conn = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
156 self._lock.acquire() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
157 try: |
5915
d0576d065993
Prefer i in d over d.has_key(i)
Christian Ebert <blacktrash@gmx.net>
parents:
4026
diff
changeset
|
158 if host in self._hostmap: |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
159 for c in self._hostmap[host]: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
160 if self._readymap[c]: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
161 self._readymap[c] = 0 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
162 conn = c |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
163 break |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
164 finally: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
165 self._lock.release() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
166 return conn |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
167 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
168 def get_all(self, host=None): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
169 if host: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
170 return list(self._hostmap.get(host, [])) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
171 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
172 return dict(self._hostmap) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
173 |
14764
a7d5816087a9
classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents:
14494
diff
changeset
|
174 class KeepAliveHandler(object): |
40043
6509fcec830c
url: allow to configure timeout on http connection
Cédric Krier <ced@b2ck.com>
parents:
40033
diff
changeset
|
175 def __init__(self, timeout=None): |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
176 self._cm = ConnectionManager() |
40043
6509fcec830c
url: allow to configure timeout on http connection
Cédric Krier <ced@b2ck.com>
parents:
40033
diff
changeset
|
177 self._timeout = timeout |
40032
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
178 self.requestscount = 0 |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
179 self.sentbytescount = 0 |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
180 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
181 #### Connection Management |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
182 def open_connections(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
183 """return a list of connected hosts and the number of connections |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
184 to each. [('foo.com:80', 2), ('bar.org', 1)]""" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
185 return [(host, len(li)) for (host, li) in self._cm.get_all().items()] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
186 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
187 def close_connection(self, host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
188 """close connection(s) to <host> |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
189 host is the host:port spec, as in 'www.cnn.com:8080' as passed in. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
190 no error occurs if there is no connection to that host.""" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
191 for h in self._cm.get_all(host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
192 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
193 h.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
194 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
195 def close_all(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
196 """close all open connections""" |
7622
4dd7b28003d2
use dict.iteritems() rather than dict.items()
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
6470
diff
changeset
|
197 for host, conns in self._cm.get_all().iteritems(): |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
198 for h in conns: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
199 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
200 h.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
201 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
202 def _request_closed(self, request, host, connection): |
17251
98166640b356
help: fix some instances of 'the the'
Mads Kiilerich <mads@kiilerich.com>
parents:
16705
diff
changeset
|
203 """tells us that this request is now closed and that the |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
204 connection is ready for another request""" |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
205 self._cm.set_ready(connection, 1) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
206 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
207 def _remove_connection(self, host, connection, close=0): |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
208 if close: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
209 connection.close() |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
210 self._cm.remove(connection) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
211 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
212 #### Transaction Execution |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
213 def http_open(self, req): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
214 return self.do_open(HTTPConnection, req) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
215 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
216 def do_open(self, http_class, req): |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
217 host = urllibcompat.gethost(req) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
218 if not host: |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
219 raise urlerr.urlerror('no host given') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
220 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
221 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
222 h = self._cm.get_ready_conn(host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
223 while h: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
224 r = self._reuse_connection(h, req, host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
225 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
226 # if this response is non-None, then it worked and we're |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
227 # done. Break out, skipping the else block. |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
228 if r: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
229 break |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
230 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
231 # connection is bad - possibly closed by server |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
232 # discard it and ask for the next free connection |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
233 h.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
234 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
235 h = self._cm.get_ready_conn(host) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
236 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
237 # no (working) free connections were found. Create a new one. |
40043
6509fcec830c
url: allow to configure timeout on http connection
Cédric Krier <ced@b2ck.com>
parents:
40033
diff
changeset
|
238 h = http_class(host, timeout=self._timeout) |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
239 if DEBUG: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
240 DEBUG.info("creating new connection to %s (%d)", |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
241 host, id(h)) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
242 self._cm.add(host, h, 0) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
243 self._start_transaction(h, req) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
244 r = h.getresponse() |
34308
9bd003052d55
keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32668
diff
changeset
|
245 # The string form of BadStatusLine is the status line. Add some context |
9bd003052d55
keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32668
diff
changeset
|
246 # to make the error message slightly more useful. |
9bd003052d55
keepalive: add more context to bad status line errors
Gregory Szorc <gregory.szorc@gmail.com>
parents:
32668
diff
changeset
|
247 except httplib.BadStatusLine as err: |
34427
a454123f5d94
keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents:
34331
diff
changeset
|
248 raise urlerr.urlerror( |
a454123f5d94
keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents:
34331
diff
changeset
|
249 _('bad HTTP status line: %s') % pycompat.sysbytes(err.line)) |
25660
328739ea70c3
global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents:
22958
diff
changeset
|
250 except (socket.error, httplib.HTTPException) as err: |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
251 raise urlerr.urlerror(err) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
252 |
39649
d6d094259d9c
keepalive: work around slight deficiency in vcr
Augie Fackler <raf@durin42.com>
parents:
37688
diff
changeset
|
253 # If not a persistent connection, don't try to reuse it. Look |
d6d094259d9c
keepalive: work around slight deficiency in vcr
Augie Fackler <raf@durin42.com>
parents:
37688
diff
changeset
|
254 # for this using getattr() since vcr doesn't define this |
d6d094259d9c
keepalive: work around slight deficiency in vcr
Augie Fackler <raf@durin42.com>
parents:
37688
diff
changeset
|
255 # attribute, and in that case always close the connection. |
d6d094259d9c
keepalive: work around slight deficiency in vcr
Augie Fackler <raf@durin42.com>
parents:
37688
diff
changeset
|
256 if getattr(r, r'will_close', True): |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
257 self._cm.remove(h) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
258 |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
259 if DEBUG: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
260 DEBUG.info("STATUS: %s, %s", r.status, r.reason) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
261 r._handler = self |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
262 r._host = host |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
263 r._url = req.get_full_url() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
264 r._connection = h |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
265 r.code = r.status |
2444
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
266 r.headers = r.msg |
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
267 r.msg = r.reason |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
268 |
30487
88a448a12ae8
keepalive: discard legacy Python support for error handling
Augie Fackler <augie@google.com>
parents:
30473
diff
changeset
|
269 return r |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
270 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
271 def _reuse_connection(self, h, req, host): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
272 """start the transaction with a re-used connection |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
273 return a response object (r) upon success or None on failure. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
274 This DOES not close or remove bad connections in cases where |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
275 it returns. However, if an unexpected exception occurs, it |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
276 will close and remove the connection before re-raising. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
277 """ |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
278 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
279 self._start_transaction(h, req) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
280 r = h.getresponse() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
281 # note: just because we got something back doesn't mean it |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
282 # worked. We'll check the version below, too. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
283 except (socket.error, httplib.HTTPException): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
284 r = None |
16705
c2d9ef43ff6c
check-code: ignore naked excepts with a "re-raise" comment
Brodie Rao <brodie@sf.io>
parents:
16688
diff
changeset
|
285 except: # re-raises |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
286 # adding this block just in case we've missed |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
287 # something we will still raise the exception, but |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
288 # lets try and close the connection and remove it |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
289 # first. We previously got into a nasty loop |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
290 # where an exception was uncaught, and so the |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
291 # connection stayed open. On the next try, the |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
17251
diff
changeset
|
292 # same exception was raised, etc. The trade-off is |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
293 # that it's now possible this call will raise |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
294 # a DIFFERENT exception |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
295 if DEBUG: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
296 DEBUG.error("unexpected exception - closing " |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
297 "connection to %s (%d)", host, id(h)) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
298 self._cm.remove(h) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
299 h.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
300 raise |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
301 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
302 if r is None or r.version == 9: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
303 # httplib falls back to assuming HTTP 0.9 if it gets a |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
304 # bad header back. This is most likely to happen if |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
305 # the socket has been closed by the server since we |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
306 # last used the connection. |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
307 if DEBUG: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
308 DEBUG.info("failed to re-use connection to %s (%d)", |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
309 host, id(h)) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
310 r = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
311 else: |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
312 if DEBUG: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
313 DEBUG.info("re-using connection to %s (%d)", host, id(h)) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
314 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
315 return r |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
316 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
317 def _start_transaction(self, h, req): |
40032
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
318 oldbytescount = h.sentbytescount |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
319 |
8233
655c435efe92
keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents:
8146
diff
changeset
|
320 # What follows mostly reimplements HTTPConnection.request() |
31999
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
321 # except it adds self.parent.addheaders in the mix and sends headers |
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
322 # in a deterministic order (to make testing easier). |
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
323 headers = util.sortdict(self.parent.addheaders) |
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
324 headers.update(sorted(req.headers.items())) |
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
325 headers.update(sorted(req.unredirected_hdrs.items())) |
aa836f56c3cc
keepalive: send HTTP request headers in a deterministic order
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30922
diff
changeset
|
326 headers = util.sortdict((n.lower(), v) for n, v in headers.items()) |
8233
655c435efe92
keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents:
8146
diff
changeset
|
327 skipheaders = {} |
37665
83250442dc81
py3: use str variables to check keys in request header
Pulkit Goyal <7895pulkit@gmail.com>
parents:
37587
diff
changeset
|
328 for n in (r'host', r'accept-encoding'): |
8233
655c435efe92
keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents:
8146
diff
changeset
|
329 if n in headers: |
37665
83250442dc81
py3: use str variables to check keys in request header
Pulkit Goyal <7895pulkit@gmail.com>
parents:
37587
diff
changeset
|
330 skipheaders[r'skip_' + n.replace(r'-', r'_')] = 1 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
331 try: |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
332 if urllibcompat.hasdata(req): |
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
333 data = urllibcompat.getdata(req) |
30922
1beeb5185930
keepalive: honor urllib2 style get_method overrides
John Mulligan <phlogistonjohn@asynchrono.us>
parents:
30688
diff
changeset
|
334 h.putrequest( |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
335 req.get_method(), urllibcompat.getselector(req), |
37688
e266e75d77dc
keepalive: add ** overlooked in 83250442dc81
Augie Fackler <augie@google.com>
parents:
37665
diff
changeset
|
336 **skipheaders) |
36295
19a04ca90413
keepalive: headers are native strings, mark them as such
Augie Fackler <augie@google.com>
parents:
35582
diff
changeset
|
337 if r'content-type' not in headers: |
19a04ca90413
keepalive: headers are native strings, mark them as such
Augie Fackler <augie@google.com>
parents:
35582
diff
changeset
|
338 h.putheader(r'Content-type', |
19a04ca90413
keepalive: headers are native strings, mark them as such
Augie Fackler <augie@google.com>
parents:
35582
diff
changeset
|
339 r'application/x-www-form-urlencoded') |
19a04ca90413
keepalive: headers are native strings, mark them as such
Augie Fackler <augie@google.com>
parents:
35582
diff
changeset
|
340 if r'content-length' not in headers: |
19a04ca90413
keepalive: headers are native strings, mark them as such
Augie Fackler <augie@google.com>
parents:
35582
diff
changeset
|
341 h.putheader(r'Content-length', r'%d' % len(data)) |
8146
4f13ed6ee544
keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents:
7875
diff
changeset
|
342 else: |
30922
1beeb5185930
keepalive: honor urllib2 style get_method overrides
John Mulligan <phlogistonjohn@asynchrono.us>
parents:
30688
diff
changeset
|
343 h.putrequest( |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
344 req.get_method(), urllibcompat.getselector(req), |
37688
e266e75d77dc
keepalive: add ** overlooked in 83250442dc81
Augie Fackler <augie@google.com>
parents:
37665
diff
changeset
|
345 **skipheaders) |
28278
b1b22185c764
keepalive: remove useless parentheses around exception type
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27616
diff
changeset
|
346 except socket.error as err: |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
347 raise urlerr.urlerror(err) |
8233
655c435efe92
keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents:
8146
diff
changeset
|
348 for k, v in headers.items(): |
8146
4f13ed6ee544
keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents:
7875
diff
changeset
|
349 h.putheader(k, v) |
4f13ed6ee544
keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents:
7875
diff
changeset
|
350 h.endheaders() |
34466
1232f7fa00c3
cleanup: use urllibcompat for renamed methods on urllib request objects
Augie Fackler <augie@google.com>
parents:
34435
diff
changeset
|
351 if urllibcompat.hasdata(req): |
8146
4f13ed6ee544
keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents:
7875
diff
changeset
|
352 h.send(data) |
4f13ed6ee544
keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents:
7875
diff
changeset
|
353 |
40032
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
354 # This will fail to record events in case of I/O failure. That's OK. |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
355 self.requestscount += 1 |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
356 self.sentbytescount += h.sentbytescount - oldbytescount |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
357 |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
358 try: |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
359 self.parent.requestscount += 1 |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
360 self.parent.sentbytescount += h.sentbytescount - oldbytescount |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
361 except AttributeError: |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
362 pass |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
363 |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
364 class HTTPHandler(KeepAliveHandler, urlreq.httphandler): |
5983
6f1fcbc58efa
httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
4026
diff
changeset
|
365 pass |
6f1fcbc58efa
httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
4026
diff
changeset
|
366 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
367 class HTTPResponse(httplib.HTTPResponse): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
368 # we need to subclass HTTPResponse in order to |
37297
97eedbd5a56c
keepalive: implement readinto()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37119
diff
changeset
|
369 # 1) add readline(), readlines(), and readinto() methods |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
370 # 2) add close_connection() methods |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
371 # 3) add info() and geturl() methods |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
372 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
373 # in order to add readline(), read must be modified to deal with a |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
374 # buffer. example: readline must read a buffer and then spit back |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
375 # one line at a time. The only real alternative is to read one |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
376 # BYTE at a time (ick). Once something has been read, it can't be |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
377 # put back (ok, maybe it can, but that's even uglier than this), |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
378 # so if you THEN do a normal read, you must first take stuff from |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
379 # the buffer. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
380 |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
17251
diff
changeset
|
381 # the read method wraps the original to accommodate buffering, |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
382 # although read() never adds to the buffer. |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
383 # Both readline and readlines have been stolen with almost no |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
384 # modification from socket.py |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
385 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
386 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
387 def __init__(self, sock, debuglevel=0, strict=0, method=None): |
34427
a454123f5d94
keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents:
34331
diff
changeset
|
388 extrakw = {} |
a454123f5d94
keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents:
34331
diff
changeset
|
389 if not pycompat.ispy3: |
35364
03112a2c9c83
py3: handle keyword arguments correctly in keepalive.py
Pulkit Goyal <7895pulkit@gmail.com>
parents:
34466
diff
changeset
|
390 extrakw[r'strict'] = True |
03112a2c9c83
py3: handle keyword arguments correctly in keepalive.py
Pulkit Goyal <7895pulkit@gmail.com>
parents:
34466
diff
changeset
|
391 extrakw[r'buffering'] = True |
32667
2806c7bbcb5e
keepalive: pass the correct arguments to HTTPResponse
Kyle Lippincott <spectral@google.com>
parents:
31999
diff
changeset
|
392 httplib.HTTPResponse.__init__(self, sock, debuglevel=debuglevel, |
34427
a454123f5d94
keepalive: python 3 portability tweaks
Augie Fackler <augie@google.com>
parents:
34331
diff
changeset
|
393 method=method, **extrakw) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
394 self.fileno = sock.fileno |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
395 self.code = None |
40033
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
396 self.receivedbytescount = 0 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
397 self._rbuf = '' |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
398 self._rbufsize = 8096 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
399 self._handler = None # inserted by the handler later |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
400 self._host = None # (same) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
401 self._url = None # (same) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
402 self._connection = None # (same) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
403 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
404 _raw_read = httplib.HTTPResponse.read |
37587
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
405 _raw_readinto = getattr(httplib.HTTPResponse, 'readinto', None) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
406 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
407 def close(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
408 if self.fp: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
409 self.fp.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
410 self.fp = None |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
411 if self._handler: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
412 self._handler._request_closed(self, self._host, |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
413 self._connection) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
414 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
415 def close_connection(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
416 self._handler._remove_connection(self._host, self._connection, close=1) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
417 self.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
418 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
419 def info(self): |
2444
5eb02f9ed804
Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents:
2435
diff
changeset
|
420 return self.headers |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
421 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
422 def geturl(self): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
423 return self._url |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
424 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
425 def read(self, amt=None): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
426 # the _rbuf test is only in this first if for speed. It's not |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
427 # logically necessary |
34331
531332502568
style: always use `x is not None` instead of `not x is None`
Alex Gaynor <agaynor@mozilla.com>
parents:
34308
diff
changeset
|
428 if self._rbuf and amt is not None: |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
429 L = len(self._rbuf) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
430 if amt > L: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
431 amt -= L |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
432 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
433 s = self._rbuf[:amt] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
434 self._rbuf = self._rbuf[amt:] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
435 return s |
39816
1cf1680b0554
keepalive: be more careful about self._rbuf when calling super impls
Augie Fackler <augie@google.com>
parents:
39649
diff
changeset
|
436 # Careful! http.client.HTTPResponse.read() on Python 3 is |
1cf1680b0554
keepalive: be more careful about self._rbuf when calling super impls
Augie Fackler <augie@google.com>
parents:
39649
diff
changeset
|
437 # implemented using readinto(), which can duplicate self._rbuf |
1cf1680b0554
keepalive: be more careful about self._rbuf when calling super impls
Augie Fackler <augie@google.com>
parents:
39649
diff
changeset
|
438 # if it's not empty. |
1cf1680b0554
keepalive: be more careful about self._rbuf when calling super impls
Augie Fackler <augie@google.com>
parents:
39649
diff
changeset
|
439 s = self._rbuf |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
440 self._rbuf = '' |
40033
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
441 data = self._raw_read(amt) |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
442 |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
443 self.receivedbytescount += len(data) |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
444 self._connection.receivedbytescount += len(data) |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
445 try: |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
446 self._handler.parent.receivedbytescount += len(data) |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
447 except AttributeError: |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
448 pass |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
449 |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
450 s += data |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
451 return s |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
452 |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
453 # stolen from Python SVN #68532 to fix issue1088 |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
454 def _read_chunked(self, amt): |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
455 chunk_left = self.chunk_left |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
456 parts = [] |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
457 |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
458 while True: |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
459 if chunk_left is None: |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
460 line = self.fp.readline() |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
461 i = line.find(';') |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
462 if i >= 0: |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
463 line = line[:i] # strip chunk-extensions |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
464 try: |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
465 chunk_left = int(line, 16) |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
466 except ValueError: |
17424
e7cfe3587ea4
fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents:
17251
diff
changeset
|
467 # close the connection as protocol synchronization is |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
468 # probably lost |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
469 self.close() |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
470 raise httplib.IncompleteRead(''.join(parts)) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
471 if chunk_left == 0: |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
472 break |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
473 if amt is None: |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
474 parts.append(self._safe_read(chunk_left)) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
475 elif amt < chunk_left: |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
476 parts.append(self._safe_read(amt)) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
477 self.chunk_left = chunk_left - amt |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
478 return ''.join(parts) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
479 elif amt == chunk_left: |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
480 parts.append(self._safe_read(amt)) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
481 self._safe_read(2) # toss the CRLF at the end of the chunk |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
482 self.chunk_left = None |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
483 return ''.join(parts) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
484 else: |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
485 parts.append(self._safe_read(chunk_left)) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
486 amt -= chunk_left |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
487 |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
488 # we read the whole chunk, get another |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
489 self._safe_read(2) # toss the CRLF at the end of the chunk |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
490 chunk_left = None |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
491 |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
492 # read and discard trailer up to the CRLF terminator |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
493 ### note: we shouldn't have any trailers! |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
494 while True: |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
495 line = self.fp.readline() |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
496 if not line: |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
497 # a vanishingly small number of sites EOF without |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
498 # sending the trailer |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
499 break |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
500 if line == '\r\n': |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
501 break |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
502 |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
503 # we read everything; close the "file" |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
504 self.close() |
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
505 |
30686
8352c42a0a0d
keepalive: don't concatenate strings when reading chunked transfer
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30487
diff
changeset
|
506 return ''.join(parts) |
7781
a45206455d85
keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents:
7622
diff
changeset
|
507 |
30687
5d06f6b73a57
keepalive: remove limit argument from readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30686
diff
changeset
|
508 def readline(self): |
30688
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
509 # Fast path for a line is already available in read buffer. |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
510 i = self._rbuf.find('\n') |
30688
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
511 if i >= 0: |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
512 i += 1 |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
513 line = self._rbuf[:i] |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
514 self._rbuf = self._rbuf[i:] |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
515 return line |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
516 |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
517 # No newline in local buffer. Read until we find one. |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
518 chunks = [self._rbuf] |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
519 i = -1 |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
520 readsize = self._rbufsize |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
521 while True: |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
522 new = self._raw_read(readsize) |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
523 if not new: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
524 break |
30688
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
525 |
40033
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
526 self.receivedbytescount += len(new) |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
527 self._connection.receivedbytescount += len(new) |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
528 try: |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
529 self._handler.parent.receivedbytescount += len(new) |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
530 except AttributeError: |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
531 pass |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
532 |
30688
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
533 chunks.append(new) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
534 i = new.find('\n') |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
535 if i >= 0: |
30688
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
536 break |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
537 |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
538 # We either have exhausted the stream or have a newline in chunks[-1]. |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
539 |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
540 # EOF |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
541 if i == -1: |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
542 self._rbuf = '' |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
543 return ''.join(chunks) |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
544 |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
545 i += 1 |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
546 self._rbuf = chunks[-1][i:] |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
547 chunks[-1] = chunks[-1][:i] |
dc5b594f41e9
keepalive: rewrite readline()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
30687
diff
changeset
|
548 return ''.join(chunks) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
549 |
19872
681f7b9213a4
check-code: check for spaces around = for named parameters
Mads Kiilerich <madski@unity3d.com>
parents:
17700
diff
changeset
|
550 def readlines(self, sizehint=0): |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
551 total = 0 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
552 list = [] |
14494
1ffeeb91c55d
check-code: flag 0/1 used as constant Boolean expression
Martin Geisler <mg@lazybytes.net>
parents:
10394
diff
changeset
|
553 while True: |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
554 line = self.readline() |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
555 if not line: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
556 break |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
557 list.append(line) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
558 total += len(line) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
559 if sizehint and total >= sizehint: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
560 break |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
561 return list |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
562 |
37297
97eedbd5a56c
keepalive: implement readinto()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37119
diff
changeset
|
563 def readinto(self, dest): |
37587
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
564 if self._raw_readinto is None: |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
565 res = self.read(len(dest)) |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
566 if not res: |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
567 return 0 |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
568 dest[0:len(res)] = res |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
569 return len(res) |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
570 total = len(dest) |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
571 have = len(self._rbuf) |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
572 if have >= total: |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
573 dest[0:total] = self._rbuf[:total] |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
574 self._rbuf = self._rbuf[total:] |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
575 return total |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
576 mv = memoryview(dest) |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
577 got = self._raw_readinto(mv[have:total]) |
40033
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
578 |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
579 self.receivedbytescount += got |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
580 self._connection.receivedbytescount += got |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
581 try: |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
582 self._handler.receivedbytescount += got |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
583 except AttributeError: |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
584 pass |
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
585 |
37587
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
586 dest[0:have] = self._rbuf |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
587 got += len(self._rbuf) |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
588 self._rbuf = '' |
192b7ad06932
keepalive: rewrite readinto() to not use read()
Augie Fackler <augie@google.com>
parents:
37297
diff
changeset
|
589 return got |
37297
97eedbd5a56c
keepalive: implement readinto()
Gregory Szorc <gregory.szorc@gmail.com>
parents:
37119
diff
changeset
|
590 |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
591 def safesend(self, str): |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
592 """Send `str' to the server. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
593 |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
594 Shamelessly ripped off from httplib to patch a bad behavior. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
595 """ |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
596 # _broken_pipe_resp is an attribute we set in this function |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
597 # if the socket is closed while we're sending data but |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
598 # the server sent us a response before hanging up. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
599 # In that case, we want to pretend to send the rest of the |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
600 # outgoing data, and then let the user use getresponse() |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
601 # (which we wrap) to get this last response before |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
602 # opening a new socket. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
603 if getattr(self, '_broken_pipe_resp', None) is not None: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
604 return |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
605 |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
606 if self.sock is None: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
607 if self.auto_open: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
608 self.connect() |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
609 else: |
16687
e34106fa0dc3
cleanup: "raise SomeException()" -> "raise SomeException"
Brodie Rao <brodie@sf.io>
parents:
16686
diff
changeset
|
610 raise httplib.NotConnected |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
611 |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
612 # send the data to the server. if we get a broken pipe, then close |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
613 # the socket. we want to reconnect when somebody tries to send again. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
614 # |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
615 # NOTE: we DO propagate the error, though, because we cannot simply |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
616 # ignore the error... the caller will know if they can retry. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
617 if self.debuglevel > 0: |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
618 print("send:", repr(str)) |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
619 try: |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
620 blocksize = 8192 |
14958
fd246aefedd3
keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents:
14764
diff
changeset
|
621 read = getattr(str, 'read', None) |
fd246aefedd3
keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents:
14764
diff
changeset
|
622 if read is not None: |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
623 if self.debuglevel > 0: |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
624 print("sending a read()able") |
14958
fd246aefedd3
keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents:
14764
diff
changeset
|
625 data = read(blocksize) |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
626 while data: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
627 self.sock.sendall(data) |
40032
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
628 self.sentbytescount += len(data) |
14958
fd246aefedd3
keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents:
14764
diff
changeset
|
629 data = read(blocksize) |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
630 else: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
631 self.sock.sendall(str) |
40032
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
632 self.sentbytescount += len(str) |
25660
328739ea70c3
global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents:
22958
diff
changeset
|
633 except socket.error as v: |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
634 reraise = True |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
635 if v[0] == errno.EPIPE: # Broken pipe |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
636 if self._HTTPConnection__state == httplib._CS_REQ_SENT: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
637 self._broken_pipe_resp = None |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
638 self._broken_pipe_resp = self.getresponse() |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
639 reraise = False |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
640 self.close() |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
641 if reraise: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
642 raise |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
643 |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
644 def wrapgetresponse(cls): |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
645 """Wraps getresponse in cls with a broken-pipe sane version. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
646 """ |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
647 def safegetresponse(self): |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
648 # In safesend() we might set the _broken_pipe_resp |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
649 # attribute, in which case the socket has already |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
650 # been closed and we just need to give them the response |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
651 # back. Otherwise, we use the normal response path. |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
652 r = getattr(self, '_broken_pipe_resp', None) |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
653 if r is not None: |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
654 return r |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
655 return cls.getresponse(self) |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
656 safegetresponse.__doc__ = cls.getresponse.__doc__ |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
657 return safegetresponse |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
658 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
659 class HTTPConnection(httplib.HTTPConnection): |
40031
f2dffa1359c6
url: have httpsconnection inherit from our custom HTTPConnection
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39816
diff
changeset
|
660 # url.httpsconnection inherits from this. So when adding/removing |
f2dffa1359c6
url: have httpsconnection inherit from our custom HTTPConnection
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39816
diff
changeset
|
661 # attributes, be sure to audit httpsconnection() for unintended |
f2dffa1359c6
url: have httpsconnection inherit from our custom HTTPConnection
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39816
diff
changeset
|
662 # consequences. |
f2dffa1359c6
url: have httpsconnection inherit from our custom HTTPConnection
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39816
diff
changeset
|
663 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
664 # use the modified response class |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
665 response_class = HTTPResponse |
9726
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
666 send = safesend |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
667 getresponse = wrapgetresponse(httplib.HTTPConnection) |
430e59ff3437
keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents:
8296
diff
changeset
|
668 |
40032
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
669 def __init__(self, *args, **kwargs): |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
670 httplib.HTTPConnection.__init__(self, *args, **kwargs) |
dc82ad1b7f77
keepalive: track request count and bytes sent
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40031
diff
changeset
|
671 self.sentbytescount = 0 |
40033
5e5b06087ec5
keepalive: track number of bytes received from an HTTP response
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40032
diff
changeset
|
672 self.receivedbytescount = 0 |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
673 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
674 ######################################################################### |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
675 ##### TEST FUNCTIONS |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
676 ######################################################################### |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
677 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
678 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
679 def continuity(url): |
29341
0d83ad967bf8
cleanup: replace uses of util.(md5|sha1|sha256|sha512) with hashlib.\1
Augie Fackler <raf@durin42.com>
parents:
28883
diff
changeset
|
680 md5 = hashlib.md5 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
681 format = '%25s: %s' |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
682 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
683 # first fetch the file with the normal http handler |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
684 opener = urlreq.buildopener() |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
685 urlreq.installopener(opener) |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
686 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
687 foo = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
688 fo.close() |
22505
232d437af120
keepalive: fix how md5 is used
Mike Hommey <mh@glandium.org>
parents:
19872
diff
changeset
|
689 m = md5(foo) |
35582
72b91f905065
py3: use node.hex(h.digest()) instead of h.hexdigest()
Pulkit Goyal <7895pulkit@gmail.com>
parents:
35364
diff
changeset
|
690 print(format % ('normal urllib', node.hex(m.digest()))) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
691 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
692 # now install the keepalive handler and try again |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
693 opener = urlreq.buildopener(HTTPHandler()) |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
694 urlreq.installopener(opener) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
695 |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
696 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
697 foo = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
698 fo.close() |
22505
232d437af120
keepalive: fix how md5 is used
Mike Hommey <mh@glandium.org>
parents:
19872
diff
changeset
|
699 m = md5(foo) |
35582
72b91f905065
py3: use node.hex(h.digest()) instead of h.hexdigest()
Pulkit Goyal <7895pulkit@gmail.com>
parents:
35364
diff
changeset
|
700 print(format % ('keepalive read', node.hex(m.digest()))) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
701 |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
702 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
703 foo = '' |
14494
1ffeeb91c55d
check-code: flag 0/1 used as constant Boolean expression
Martin Geisler <mg@lazybytes.net>
parents:
10394
diff
changeset
|
704 while True: |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
705 f = fo.readline() |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
706 if f: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
707 foo = foo + f |
34435
5326e4ef1dab
style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents:
34427
diff
changeset
|
708 else: |
5326e4ef1dab
style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents:
34427
diff
changeset
|
709 break |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
710 fo.close() |
22505
232d437af120
keepalive: fix how md5 is used
Mike Hommey <mh@glandium.org>
parents:
19872
diff
changeset
|
711 m = md5(foo) |
35582
72b91f905065
py3: use node.hex(h.digest()) instead of h.hexdigest()
Pulkit Goyal <7895pulkit@gmail.com>
parents:
35364
diff
changeset
|
712 print(format % ('keepalive readline', node.hex(m.digest()))) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
713 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
714 def comp(N, url): |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
715 print(' making %i connections to:\n %s' % (N, url)) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
716 |
37119
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
717 procutil.stdout.write(' first using the normal urllib handlers') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
718 # first use normal opener |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
719 opener = urlreq.buildopener() |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
720 urlreq.installopener(opener) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
721 t1 = fetch(N, url) |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
722 print(' TIME: %.3f s' % t1) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
723 |
37119
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
724 procutil.stdout.write(' now using the keepalive handler ') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
725 # now install the keepalive handler and try again |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
726 opener = urlreq.buildopener(HTTPHandler()) |
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
727 urlreq.installopener(opener) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
728 t2 = fetch(N, url) |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
729 print(' TIME: %.3f s' % t2) |
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
730 print(' improvement factor: %.2f' % (t1 / t2)) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
731 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
732 def fetch(N, url, delay=0): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
733 import time |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
734 lens = [] |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
735 starttime = time.time() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
736 for i in range(N): |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
737 if delay and i > 0: |
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
738 time.sleep(delay) |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
739 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
740 foo = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
741 fo.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
742 lens.append(len(foo)) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
743 diff = time.time() - starttime |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
744 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
745 j = 0 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
746 for i in lens[1:]: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
747 j = j + 1 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
748 if not i == lens[0]: |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
749 print("WARNING: inconsistent length on read %i: %i" % (j, i)) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
750 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
751 return diff |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
752 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
753 def test_timeout(url): |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
754 global DEBUG |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
755 dbbackup = DEBUG |
14764
a7d5816087a9
classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents:
14494
diff
changeset
|
756 class FakeLogger(object): |
10282
08a0f04b56bd
many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents:
9726
diff
changeset
|
757 def debug(self, msg, *args): |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
758 print(msg % args) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
759 info = warning = error = debug |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
760 DEBUG = FakeLogger() |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
761 print(" fetching the file to establish a connection") |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
762 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
763 data1 = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
764 fo.close() |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
765 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
766 i = 20 |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
767 print(" waiting %i seconds for the server to close the connection" % i) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
768 while i > 0: |
37119
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
769 procutil.stdout.write('\r %2i' % i) |
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
770 procutil.stdout.flush() |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
771 time.sleep(1) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
772 i -= 1 |
37119
d4a2e0d5d042
procutil: bulk-replace util.std* to point to new module
Yuya Nishihara <yuya@tcha.org>
parents:
36295
diff
changeset
|
773 procutil.stderr.write('\r') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
774 |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
775 print(" fetching the file a second time") |
28883
032c4c2f802a
pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents:
28278
diff
changeset
|
776 fo = urlreq.urlopen(url) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
777 data2 = fo.read() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
778 fo.close() |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
779 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
780 if data1 == data2: |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
781 print(' data are identical') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
782 else: |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
783 print(' ERROR: DATA DIFFER') |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
784 |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
785 DEBUG = dbbackup |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
786 |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
787 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
788 def test(url, N=10): |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
789 print("performing continuity test (making sure stuff isn't corrupted)") |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
790 continuity(url) |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
791 print('') |
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
792 print("performing speed comparison") |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
793 comp(N, url) |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
794 print('') |
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
795 print("performing dropped-connection check") |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
796 test_timeout(url) |
2600
c4325f0a9b91
clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2444
diff
changeset
|
797 |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
798 if __name__ == '__main__': |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
799 import time |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
800 try: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
801 N = int(sys.argv[1]) |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
802 url = sys.argv[2] |
16688
cfb6682961b8
cleanup: replace naked excepts with more specific ones
Brodie Rao <brodie@sf.io>
parents:
16687
diff
changeset
|
803 except (IndexError, ValueError): |
27616
0765d8423fbc
keepalive: use print function
Gregory Szorc <gregory.szorc@gmail.com>
parents:
27507
diff
changeset
|
804 print("%s <integer> <url>" % sys.argv[0]) |
2435
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
805 else: |
ff2bac730b99
http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff
changeset
|
806 test(url, N) |