mercurial/httpconnection.py
author Pierre-Yves David <pierre-yves.david@octobus.net>
Fri, 20 Nov 2020 11:22:28 +0100
changeset 45986 f9f8d8aa9a92
parent 44298 0e8b28fb751b
child 46819 d4ba4d51f85f
permissions -rw-r--r--
copies: clarify the return of _merge_copies_dict I misused that function twice in the past few days, so lets clarify the API. Differential Revision: https://phab.mercurial-scm.org/D9418
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
     1
# httpconnection.py - urllib2 handler for new http support
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
     2
#
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
     3
# Copyright 2005, 2006, 2007, 2008 Matt Mackall <mpm@selenic.com>
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
     4
# Copyright 2006, 2007 Alexis S. L. Carvalho <alexis@cecm.usp.br>
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
     5
# Copyright 2006 Vadim Gelfer <vadim.gelfer@gmail.com>
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
     6
# Copyright 2011 Google, Inc.
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
     7
#
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
     8
# This software may be used and distributed according to the terms of the
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
     9
# GNU General Public License version 2 or any later version.
27521
b1adf32b0605 httpconnection: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26347
diff changeset
    10
b1adf32b0605 httpconnection: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26347
diff changeset
    11
from __future__ import absolute_import
b1adf32b0605 httpconnection: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26347
diff changeset
    12
b1adf32b0605 httpconnection: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26347
diff changeset
    13
import os
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    14
27521
b1adf32b0605 httpconnection: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26347
diff changeset
    15
from .i18n import _
43085
eef9a2d67051 py3: manually import pycompat.open into files that need it
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43077
diff changeset
    16
from .pycompat import open
27521
b1adf32b0605 httpconnection: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26347
diff changeset
    17
from . import (
36651
6b1eb4c610b4 httpconnection: convert url to bytes in readauthforuri
Augie Fackler <augie@google.com>
parents: 36426
diff changeset
    18
    pycompat,
27521
b1adf32b0605 httpconnection: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26347
diff changeset
    19
    util,
b1adf32b0605 httpconnection: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26347
diff changeset
    20
)
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    21
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28526
diff changeset
    22
urlerr = util.urlerr
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28526
diff changeset
    23
urlreq = util.urlreq
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28526
diff changeset
    24
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    25
# moved here from url.py to avoid a cycle
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    26
class httpsendfile(object):
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    27
    """This is a wrapper around the objects returned by python's "open".
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    28
15152
94b200a11cf7 http: handle push of bundles > 2 GB again (issue3017)
Mads Kiilerich <mads@kiilerich.com>
parents: 15025
diff changeset
    29
    Its purpose is to send file-like objects via HTTP.
94b200a11cf7 http: handle push of bundles > 2 GB again (issue3017)
Mads Kiilerich <mads@kiilerich.com>
parents: 15025
diff changeset
    30
    It do however not define a __len__ attribute because the length
94b200a11cf7 http: handle push of bundles > 2 GB again (issue3017)
Mads Kiilerich <mads@kiilerich.com>
parents: 15025
diff changeset
    31
    might be more than Py_ssize_t can handle.
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    32
    """
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    33
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    34
    def __init__(self, ui, *args, **kwargs):
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    35
        self.ui = ui
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    36
        self._data = open(*args, **kwargs)
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    37
        self.seek = self._data.seek
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    38
        self.close = self._data.close
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    39
        self.write = self._data.write
15152
94b200a11cf7 http: handle push of bundles > 2 GB again (issue3017)
Mads Kiilerich <mads@kiilerich.com>
parents: 15025
diff changeset
    40
        self.length = os.fstat(self._data.fileno()).st_size
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    41
        self._pos = 0
44298
0e8b28fb751b httpconnection: allow `httpsendfile` subclasses to suppress the progressbar
Matt Harbison <matt_harbison@yahoo.com>
parents: 43106
diff changeset
    42
        self._progress = self._makeprogress()
0e8b28fb751b httpconnection: allow `httpsendfile` subclasses to suppress the progressbar
Matt Harbison <matt_harbison@yahoo.com>
parents: 43106
diff changeset
    43
0e8b28fb751b httpconnection: allow `httpsendfile` subclasses to suppress the progressbar
Matt Harbison <matt_harbison@yahoo.com>
parents: 43106
diff changeset
    44
    def _makeprogress(self):
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    45
        # We pass double the max for total because we currently have
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    46
        # to send the bundle twice in the case of a server that
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    47
        # requires authentication. Since we can't know until we try
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    48
        # once whether authentication will be required, just lie to
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    49
        # the user and maybe the push succeeds suddenly at 50%.
44298
0e8b28fb751b httpconnection: allow `httpsendfile` subclasses to suppress the progressbar
Matt Harbison <matt_harbison@yahoo.com>
parents: 43106
diff changeset
    50
        return self.ui.makeprogress(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
    51
            _(b'sending'), unit=_(b'kb'), total=(self.length // 1024 * 2)
43075
57875cf423c9 style: run a patched black on a subset of mercurial
Augie Fackler <augie@google.com>
parents: 41759
diff changeset
    52
        )
38393
5f9d436cd3b7 httpconnection: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 36651
diff changeset
    53
5f9d436cd3b7 httpconnection: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 36651
diff changeset
    54
    def read(self, *args, **kwargs):
5f9d436cd3b7 httpconnection: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 36651
diff changeset
    55
        ret = self._data.read(*args, **kwargs)
5f9d436cd3b7 httpconnection: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 36651
diff changeset
    56
        if not ret:
5f9d436cd3b7 httpconnection: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 36651
diff changeset
    57
            self._progress.complete()
5f9d436cd3b7 httpconnection: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 36651
diff changeset
    58
            return ret
5f9d436cd3b7 httpconnection: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 36651
diff changeset
    59
        self._pos += len(ret)
5f9d436cd3b7 httpconnection: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 36651
diff changeset
    60
        self._progress.update(self._pos // 1024)
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    61
        return ret
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    62
30142
3dcaf1c4e90d largefiles: use context for file closing
Mads Kiilerich <madski@unity3d.com>
parents: 29250
diff changeset
    63
    def __enter__(self):
3dcaf1c4e90d largefiles: use context for file closing
Mads Kiilerich <madski@unity3d.com>
parents: 29250
diff changeset
    64
        return self
3dcaf1c4e90d largefiles: use context for file closing
Mads Kiilerich <madski@unity3d.com>
parents: 29250
diff changeset
    65
3dcaf1c4e90d largefiles: use context for file closing
Mads Kiilerich <madski@unity3d.com>
parents: 29250
diff changeset
    66
    def __exit__(self, exc_type, exc_val, exc_tb):
3dcaf1c4e90d largefiles: use context for file closing
Mads Kiilerich <madski@unity3d.com>
parents: 29250
diff changeset
    67
        self.close()
3dcaf1c4e90d largefiles: use context for file closing
Mads Kiilerich <madski@unity3d.com>
parents: 29250
diff changeset
    68
43075
57875cf423c9 style: run a patched black on a subset of mercurial
Augie Fackler <augie@google.com>
parents: 41759
diff changeset
    69
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    70
# moved here from url.py to avoid a cycle
15025
0593e8f81c71 http: pass user to readauthforuri() (fix 4a43e23b8c55)
Patrick Mezard <pmezard@gmail.com>
parents: 15005
diff changeset
    71
def readauthforuri(ui, uri, user):
36651
6b1eb4c610b4 httpconnection: convert url to bytes in readauthforuri
Augie Fackler <augie@google.com>
parents: 36426
diff changeset
    72
    uri = pycompat.bytesurl(uri)
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    73
    # Read configuration
31300
0c8a042b193d httpconnection: rename config to groups
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31299
diff changeset
    74
    groups = {}
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
    75
    for key, val in ui.configitems(b'auth'):
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
    76
        if key in (b'cookiefile',):
31935
566cb89050b7 httpconnection: allow a global auth.cookiefile config entry
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31488
diff changeset
    77
            continue
566cb89050b7 httpconnection: allow a global auth.cookiefile config entry
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31488
diff changeset
    78
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
    79
        if b'.' not in key:
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
    80
            ui.warn(_(b"ignoring invalid [auth] key '%s'\n") % key)
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    81
            continue
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
    82
        group, setting = key.rsplit(b'.', 1)
31300
0c8a042b193d httpconnection: rename config to groups
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31299
diff changeset
    83
        gdict = groups.setdefault(group, {})
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
    84
        if setting in (b'username', b'cert', b'key'):
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    85
            val = util.expandpath(val)
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    86
        gdict[setting] = val
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    87
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    88
    # Find the best match
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
    89
    scheme, hostpath = uri.split(b'://', 1)
15005
4a43e23b8c55 hgweb: do not ignore [auth] if url has a username (issue2822)
Patrick Mezard <pmezard@gmail.com>
parents: 14430
diff changeset
    90
    bestuser = None
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    91
    bestlen = 0
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    92
    bestauth = None
43106
d783f945a701 py3: finish porting iteritems() to pycompat and remove source transformer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43085
diff changeset
    93
    for group, auth in pycompat.iteritems(groups):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
    94
        if user and user != auth.get(b'username', user):
15005
4a43e23b8c55 hgweb: do not ignore [auth] if url has a username (issue2822)
Patrick Mezard <pmezard@gmail.com>
parents: 14430
diff changeset
    95
            # If a username was set in the URI, the entry username
4a43e23b8c55 hgweb: do not ignore [auth] if url has a username (issue2822)
Patrick Mezard <pmezard@gmail.com>
parents: 14430
diff changeset
    96
            # must either match it or be unset
4a43e23b8c55 hgweb: do not ignore [auth] if url has a username (issue2822)
Patrick Mezard <pmezard@gmail.com>
parents: 14430
diff changeset
    97
            continue
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
    98
        prefix = auth.get(b'prefix')
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
    99
        if not prefix:
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
   100
            continue
40663
c53f0ead5781 http: allow 'auth.prefix' to have a username consistent with the URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 38393
diff changeset
   101
c53f0ead5781 http: allow 'auth.prefix' to have a username consistent with the URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 38393
diff changeset
   102
        prefixurl = util.url(prefix)
c53f0ead5781 http: allow 'auth.prefix' to have a username consistent with the URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 38393
diff changeset
   103
        if prefixurl.user and prefixurl.user != user:
c53f0ead5781 http: allow 'auth.prefix' to have a username consistent with the URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 38393
diff changeset
   104
            # If a username was set in the prefix, it must match the username in
c53f0ead5781 http: allow 'auth.prefix' to have a username consistent with the URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 38393
diff changeset
   105
            # the URI.
c53f0ead5781 http: allow 'auth.prefix' to have a username consistent with the URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 38393
diff changeset
   106
            continue
c53f0ead5781 http: allow 'auth.prefix' to have a username consistent with the URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 38393
diff changeset
   107
c53f0ead5781 http: allow 'auth.prefix' to have a username consistent with the URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 38393
diff changeset
   108
        # The URI passed in has been stripped of credentials, so erase the user
c53f0ead5781 http: allow 'auth.prefix' to have a username consistent with the URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 38393
diff changeset
   109
        # here to allow simpler matching.
c53f0ead5781 http: allow 'auth.prefix' to have a username consistent with the URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 38393
diff changeset
   110
        prefixurl.user = None
c53f0ead5781 http: allow 'auth.prefix' to have a username consistent with the URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 38393
diff changeset
   111
        prefix = bytes(prefixurl)
c53f0ead5781 http: allow 'auth.prefix' to have a username consistent with the URI
Matt Harbison <matt_harbison@yahoo.com>
parents: 38393
diff changeset
   112
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
   113
        p = prefix.split(b'://', 1)
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
   114
        if len(p) > 1:
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
   115
            schemes, prefix = [p[0]], p[1]
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
   116
        else:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
   117
            schemes = (auth.get(b'schemes') or b'https').split()
43075
57875cf423c9 style: run a patched black on a subset of mercurial
Augie Fackler <augie@google.com>
parents: 41759
diff changeset
   118
        if (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
   119
            (prefix == b'*' or hostpath.startswith(prefix))
43075
57875cf423c9 style: run a patched black on a subset of mercurial
Augie Fackler <augie@google.com>
parents: 41759
diff changeset
   120
            and (
57875cf423c9 style: run a patched black on a subset of mercurial
Augie Fackler <augie@google.com>
parents: 41759
diff changeset
   121
                len(prefix) > bestlen
57875cf423c9 style: run a patched black on a subset of mercurial
Augie Fackler <augie@google.com>
parents: 41759
diff changeset
   122
                or (
57875cf423c9 style: run a patched black on a subset of mercurial
Augie Fackler <augie@google.com>
parents: 41759
diff changeset
   123
                    len(prefix) == bestlen
57875cf423c9 style: run a patched black on a subset of mercurial
Augie Fackler <augie@google.com>
parents: 41759
diff changeset
   124
                    and not bestuser
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
   125
                    and b'username' in auth
43075
57875cf423c9 style: run a patched black on a subset of mercurial
Augie Fackler <augie@google.com>
parents: 41759
diff changeset
   126
                )
57875cf423c9 style: run a patched black on a subset of mercurial
Augie Fackler <augie@google.com>
parents: 41759
diff changeset
   127
            )
57875cf423c9 style: run a patched black on a subset of mercurial
Augie Fackler <augie@google.com>
parents: 41759
diff changeset
   128
            and scheme in schemes
57875cf423c9 style: run a patched black on a subset of mercurial
Augie Fackler <augie@google.com>
parents: 41759
diff changeset
   129
        ):
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
   130
            bestlen = len(prefix)
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
   131
            bestauth = group, auth
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
   132
            bestuser = auth.get(b'username')
15005
4a43e23b8c55 hgweb: do not ignore [auth] if url has a username (issue2822)
Patrick Mezard <pmezard@gmail.com>
parents: 14430
diff changeset
   133
            if user and not bestuser:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43075
diff changeset
   134
                auth[b'username'] = user
14244
e7525a555a64 url: use new http support if requested by the user
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
   135
    return bestauth