Mercurial > hg
view mercurial/mail.py @ 50338:81c7d04f4722 stable
match: match explicit file using a set
The matcher as all the logic to do quick comparison against explicit patterns,
however the pattern matcher was shadowing the code using that set and used the
compiled regex pattern in all cases, which is quite slow. We restore the usage
of the set based matching to boost performance.
Building the regexp is still consuming a large amount of time (actually, the
majority of the time), which is still silly. Maybe using re2 would help that,
but this is a quest for another adventure.
Another path to improve this is to have a pattern type dedicated to match the
exact path to a file only (not a directory). This pattern could use the set
matching only and be skipped in the regex all together.
Benchmarks
==========
In the following benchmark we are comparing the `hg cat` and `hg files` run
time when matching against all files in the repository. They are
run:
- without the rust extensions
- with the standard python engine (so without re2)
Performance improvement in this series
--------------------------------------
###### hg files ###############################################################
### mercurial-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset: 0.230092 seconds
prev-changeset: 0.230069 seconds
this-changeset: 0.211425 seconds (-8.36%)
### mercurial-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset: 0.234235 seconds
prev-changeset: 0.231165 seconds (-1.38%)
this-changeset: 0.212300 seconds (-9.43%)
### pypy-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset: 0.613567 seconds
prev-changeset: 0.616799 seconds
this-changeset: 0.510852 seconds (-16.82%)
### pypy-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset: 0.801880 seconds
prev-changeset: 0.616393 seconds (-23.22%)
this-changeset: 0.511903 seconds (-36.23%)
### netbeans-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset: 21.541828 seconds
prev-changeset: 21.586773 seconds
this-changeset: 13.648347 seconds (-36.76%)
### netbeans-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset: 172.759857 seconds
prev-changeset: 21.908197 seconds (-87.32%)
this-changeset: 13.945110 seconds (-91.93%)
### mozilla-central-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset: 62.474221 seconds
prev-changeset: 61.279490 seconds (-1.22%)
this-changeset: 29.529469 seconds (-52.40%)
### mozilla-central-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset: 1364.180218 seconds
prev-changeset: 62.473549 seconds (-95.40%)
this-changeset: 30.625249 seconds (-97.75%)
###### hg cat #################################################################
### mercurial-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset: 0.764407 seconds
prev-changeset: 0.763883 seconds
this-changeset: 0.737326 seconds (-3.68%)
### mercurial-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset: 0.768924 seconds
prev-changeset: 0.765848 seconds
this-changeset: 0.174d0b seconds (-4.44%)
### pypy-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset: 2.065220 seconds
prev-changeset: 2.070498 seconds
this-changeset: 1.939482 seconds (-6.08%)
### pypy-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset: 2.276388 seconds
prev-changeset: 2.069197 seconds (-9.15%)
this-changeset: 1.931746 seconds (-15.19%)
### netbeans-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset: 40.967983 seconds
prev-changeset: 41.392423 seconds
this-changeset: 32.181681 seconds (-22.20%)
### netbeans-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset: 216.388709 seconds
prev-changeset: 41.648689 seconds (-80.88%)
this-changeset: 32.580817 seconds (-85.04%)
### mozilla-central-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset: 105.228510 seconds
prev-changeset: 103.315670 seconds (-1.23%)
this-changeset: 69.416118 seconds (-33.64%)
### mozilla-central-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset: 1448.722784 seconds
prev-changeset: 104.369358 seconds (-92.80%)
this-changeset: 70.554789 seconds (-95.13%)
Different way to list the same data with this revision
------------------------------------------------------
###### hg files ###############################################################
### mercurial-2018-08-01-zstd-sparse-revlog
root: 0.119182 seconds
glob: 0.120697 seconds (+1.27%)
sorted: 0.211425 seconds (+77.40%)
shuffled: 0.212300 seconds (+78.13%)
### pypy-2018-08-01-zstd-sparse-revlog
root: 0.121986 seconds
glob: 0.124822 seconds (+2.32%)
sorted: 0.510852 seconds (+318.78%)
shuffled: 0.511903 seconds (+319.64%)
### netbeans-2018-08-01-zstd-sparse-revlog
root: 0.173984 seconds
glob: 0.227203 seconds (+30.59%)
sorted: 13.648347 seconds (+7744.59%)
shuffled: 13.945110 seconds (+7915.16%)
### mozilla-central-2018-08-01-zstd-sparse-revlog
root: 0.366463 seconds
glob: 0.491030 seconds (+33.99%)
sorted: 29.529469 seconds (+7957.96%)
shuffled: 30.625249 seconds (+8256.97%)
###### hg cat #################################################################
### mercurial-2018-08-01-zstd-sparse-revlog
glob: 0.647471 seconds
root: 0.643120 seconds
shuffled: 0.174d0b seconds (+13.92%)
sorted: 0.737326 seconds (+13.88%)
### mozilla-central-2018-08-01-zstd-sparse-revlog
glob: 40.596983 seconds
root: 40.129136 seconds
shuffled: 70.554789 seconds (+73.79%)
sorted: 69.416118 seconds (+70.99%)
### netbeans-2018-08-01-zstd-sparse-revlog
glob: 18.777924 seconds
root: 18.613905 seconds
shuffled: 32.580817 seconds (+73.51%)
sorted: 32.181681 seconds (+71.38%)
### pypy-2018-08-01-zstd-sparse-revlog
glob: 1.555319 seconds
root: 1.536534 seconds
shuffled: 1.931746 seconds (+24.20%)
sorted: 1.939482 seconds (+24.70%)
author | Pierre-Yves David <pierre-yves.david@octobus.net> |
---|---|
date | Sat, 01 Apr 2023 05:58:59 +0200 |
parents | 1d6c6ad645e1 |
children | 1c7453f2bb57 |
line wrap: on
line source
# mail.py - mail sending bits for mercurial # # Copyright 2006 Olivia Mackall <olivia@selenic.com> # # This software may be used and distributed according to the terms of the # GNU General Public License version 2 or any later version. import email import email.charset import email.generator import email.header import email.message import email.parser import io import os import smtplib import socket import time from .i18n import _ from .pycompat import ( getattr, open, ) from . import ( encoding, error, pycompat, sslutil, util, ) from .utils import ( procutil, stringutil, urlutil, ) if pycompat.TYPE_CHECKING: from typing import Any, List, Tuple, Union # keep pyflakes happy assert all((Any, List, Tuple, Union)) class STARTTLS(smtplib.SMTP): """Derived class to verify the peer certificate for STARTTLS. This class allows to pass any keyword arguments to SSL socket creation. """ def __init__(self, ui, host=None, **kwargs): smtplib.SMTP.__init__(self, **kwargs) self._ui = ui self._host = host def starttls(self, keyfile=None, certfile=None): if not self.has_extn("starttls"): msg = b"STARTTLS extension not supported by server" raise smtplib.SMTPException(msg) (resp, reply) = self.docmd("STARTTLS") if resp == 220: self.sock = sslutil.wrapsocket( self.sock, keyfile, certfile, ui=self._ui, serverhostname=self._host, ) self.file = self.sock.makefile("rb") self.helo_resp = None self.ehlo_resp = None self.esmtp_features = {} self.does_esmtp = 0 return (resp, reply) class SMTPS(smtplib.SMTP): """Derived class to verify the peer certificate for SMTPS. This class allows to pass any keyword arguments to SSL socket creation. """ def __init__(self, ui, keyfile=None, certfile=None, host=None, **kwargs): self.keyfile = keyfile self.certfile = certfile smtplib.SMTP.__init__(self, **kwargs) self._host = host self.default_port = smtplib.SMTP_SSL_PORT self._ui = ui def _get_socket(self, host, port, timeout): if self.debuglevel > 0: self._ui.debug(b'connect: %r\n' % ((host, port),)) new_socket = socket.create_connection((host, port), timeout) new_socket = sslutil.wrapsocket( new_socket, self.keyfile, self.certfile, ui=self._ui, serverhostname=self._host, ) self.file = new_socket.makefile('rb') return new_socket def _pyhastls(): # type: () -> bool """Returns true iff Python has TLS support, false otherwise.""" try: import ssl getattr(ssl, 'HAS_TLS', False) return True except ImportError: return False def _smtp(ui): '''build an smtp connection and return a function to send mail''' local_hostname = ui.config(b'smtp', b'local_hostname') tls = ui.config(b'smtp', b'tls') # backward compatible: when tls = true, we use starttls. starttls = tls == b'starttls' or stringutil.parsebool(tls) smtps = tls == b'smtps' if (starttls or smtps) and not _pyhastls(): raise error.Abort(_(b"can't use TLS: Python SSL support not installed")) mailhost = ui.config(b'smtp', b'host') if not mailhost: raise error.Abort(_(b'smtp.host not configured - cannot send mail')) if smtps: ui.note(_(b'(using smtps)\n')) s = SMTPS(ui, local_hostname=local_hostname, host=mailhost) elif starttls: s = STARTTLS(ui, local_hostname=local_hostname, host=mailhost) else: s = smtplib.SMTP(local_hostname=local_hostname) if smtps: defaultport = 465 else: defaultport = 25 mailport = urlutil.getport(ui.config(b'smtp', b'port', defaultport)) ui.note(_(b'sending mail: smtp host %s, port %d\n') % (mailhost, mailport)) s.connect(host=mailhost, port=mailport) if starttls: ui.note(_(b'(using starttls)\n')) s.ehlo() s.starttls() s.ehlo() if starttls or smtps: ui.note(_(b'(verifying remote certificate)\n')) sslutil.validatesocket(s.sock) try: _smtp_login(ui, s, mailhost, mailport) except smtplib.SMTPException as inst: raise error.Abort(stringutil.forcebytestr(inst)) def send(sender, recipients, msg): try: return s.sendmail(sender, recipients, msg) except smtplib.SMTPRecipientsRefused as inst: recipients = [r[1] for r in inst.recipients.values()] raise error.Abort(b'\n' + b'\n'.join(recipients)) except smtplib.SMTPException as inst: raise error.Abort(stringutil.forcebytestr(inst)) return send def _smtp_login(ui, smtp, mailhost, mailport): """A hook for the keyring extension to perform the actual SMTP login. An already connected SMTP object of the proper type is provided, based on the current configuration. The host and port to which the connection was established are provided for accessibility, since the SMTP object doesn't provide an accessor. ``smtplib.SMTPException`` is raised on error. """ username = ui.config(b'smtp', b'username') password = ui.config(b'smtp', b'password') if username: if password: password = encoding.strfromlocal(password) else: password = ui.getpass() if password is not None: password = encoding.strfromlocal(password) if username and password: ui.note(_(b'(authenticating to mail server as %s)\n') % username) username = encoding.strfromlocal(username) smtp.login(username, password) def _sendmail(ui, sender, recipients, msg): '''send mail using sendmail.''' program = ui.config(b'email', b'method') def stremail(x): return procutil.shellquote(stringutil.email(encoding.strtolocal(x))) cmdline = b'%s -f %s %s' % ( program, stremail(sender), b' '.join(map(stremail, recipients)), ) ui.note(_(b'sending mail: %s\n') % cmdline) fp = procutil.popen(cmdline, b'wb') fp.write(util.tonativeeol(msg)) ret = fp.close() if ret: raise error.Abort( b'%s %s' % ( os.path.basename(procutil.shellsplit(program)[0]), procutil.explainexit(ret), ) ) def _mbox(mbox, sender, recipients, msg): '''write mails to mbox''' # TODO: use python mbox library for proper locking with open(mbox, b'ab+') as fp: # Should be time.asctime(), but Windows prints 2-characters day # of month instead of one. Make them print the same thing. date = time.strftime('%a %b %d %H:%M:%S %Y', time.localtime()) fp.write( b'From %s %s\n' % (encoding.strtolocal(sender), encoding.strtolocal(date)) ) fp.write(msg) fp.write(b'\n\n') def connect(ui, mbox=None): """make a mail connection. return a function to send mail. call as sendmail(sender, list-of-recipients, msg).""" if mbox: open(mbox, b'wb').close() return lambda s, r, m: _mbox(mbox, s, r, m) if ui.config(b'email', b'method') == b'smtp': return _smtp(ui) return lambda s, r, m: _sendmail(ui, s, r, m) def sendmail(ui, sender, recipients, msg, mbox=None): send = connect(ui, mbox=mbox) return send(sender, recipients, msg) def validateconfig(ui): '''determine if we have enough config data to try sending email.''' method = ui.config(b'email', b'method') if method == b'smtp': if not ui.config(b'smtp', b'host'): raise error.Abort( _( b'smtp specified as email transport, ' b'but no smtp host configured' ) ) else: command = procutil.shellsplit(method) command = command[0] if command else b'' if not (command and procutil.findexe(command)): raise error.Abort( _(b'%r specified as email transport, but not in PATH') % command ) def codec2iana(cs): # type: (str) -> str ''' ''' cs = email.charset.Charset(cs).input_charset.lower() # "latin1" normalizes to "iso8859-1", standard calls for "iso-8859-1" if cs.startswith("iso") and not cs.startswith("iso-"): return "iso-" + cs[3:] return cs def mimetextpatch(s, subtype='plain', display=False): # type: (bytes, str, bool) -> email.message.Message """Return MIME message suitable for a patch. Charset will be detected by first trying to decode as us-ascii, then utf-8, and finally the global encodings. If all those fail, fall back to ISO-8859-1, an encoding with that allows all byte sequences. Transfer encodings will be used if necessary.""" cs = [ 'us-ascii', 'utf-8', pycompat.sysstr(encoding.encoding), pycompat.sysstr(encoding.fallbackencoding), ] if display: cs = ['us-ascii'] for charset in cs: try: s.decode(charset) return mimetextqp(s, subtype, codec2iana(charset)) except UnicodeDecodeError: pass return mimetextqp(s, subtype, "iso-8859-1") def mimetextqp(body, subtype, charset): # type: (bytes, str, str) -> email.message.Message """Return MIME message. Quoted-printable transfer encoding will be used if necessary. """ cs = email.charset.Charset(charset) msg = email.message.Message() msg.set_type('text/' + subtype) for line in body.splitlines(): if len(line) > 950: cs.body_encoding = email.charset.QP break # On Python 2, this simply assigns a value. Python 3 inspects # body and does different things depending on whether it has # encode() or decode() attributes. We can get the old behavior # if we pass a str and charset is None and we call set_charset(). # But we may get into trouble later due to Python attempting to # encode/decode using the registered charset (or attempting to # use ascii in the absence of a charset). msg.set_payload(body, cs) return msg def _charsets(ui): # type: (Any) -> List[str] '''Obtains charsets to send mail parts not containing patches.''' charsets = [ pycompat.sysstr(cs.lower()) for cs in ui.configlist(b'email', b'charsets') ] fallbacks = [ pycompat.sysstr(encoding.fallbackencoding.lower()), pycompat.sysstr(encoding.encoding.lower()), 'utf-8', ] for cs in fallbacks: # find unique charsets while keeping order if cs not in charsets: charsets.append(cs) return [cs for cs in charsets if not cs.endswith('ascii')] def _encode(ui, s, charsets): # type: (Any, bytes, List[str]) -> Tuple[bytes, str] """Returns (converted) string, charset tuple. Finds out best charset by cycling through sendcharsets in descending order. Tries both encoding and fallbackencoding for input. Only as last resort send as is in fake ascii. Caveat: Do not use for mail parts containing patches!""" sendcharsets = charsets or _charsets(ui) if not isinstance(s, bytes): # We have unicode data, which we need to try and encode to # some reasonable-ish encoding. Try the encodings the user # wants, and fall back to garbage-in-ascii. for ocs in sendcharsets: try: return s.encode(ocs), ocs except UnicodeEncodeError: pass except LookupError: ui.warn( _(b'ignoring invalid sendcharset: %s\n') % pycompat.sysbytes(ocs) ) else: # Everything failed, ascii-armor what we've got and send it. return s.encode('ascii', 'backslashreplace'), 'us-ascii' # We have a bytes of unknown encoding. We'll try and guess a valid # encoding, falling back to pretending we had ascii even though we # know that's wrong. try: s.decode('ascii') except UnicodeDecodeError: for ics in (encoding.encoding, encoding.fallbackencoding): ics = pycompat.sysstr(ics) try: u = s.decode(ics) except UnicodeDecodeError: continue for ocs in sendcharsets: try: return u.encode(ocs), ocs except UnicodeEncodeError: pass except LookupError: ui.warn( _(b'ignoring invalid sendcharset: %s\n') % pycompat.sysbytes(ocs) ) # if ascii, or all conversion attempts fail, send (broken) ascii return s, 'us-ascii' def headencode(ui, s, charsets=None, display=False): # type: (Any, Union[bytes, str], List[str], bool) -> str '''Returns RFC-2047 compliant header from given string.''' if not display: # split into words? s, cs = _encode(ui, s, charsets) return email.header.Header(s, cs).encode() return encoding.strfromlocal(s) def _addressencode(ui, name, addr, charsets=None): # type: (Any, str, str, List[str]) -> str addr = encoding.strtolocal(addr) name = headencode(ui, name, charsets) try: acc, dom = addr.split(b'@') acc.decode('ascii') dom = dom.decode(pycompat.sysstr(encoding.encoding)).encode('idna') addr = b'%s@%s' % (acc, dom) except UnicodeDecodeError: raise error.Abort(_(b'invalid email address: %s') % addr) except ValueError: try: # too strict? addr.decode('ascii') except UnicodeDecodeError: raise error.Abort(_(b'invalid local address: %s') % addr) return email.utils.formataddr((name, encoding.strfromlocal(addr))) def addressencode(ui, address, charsets=None, display=False): # type: (Any, bytes, List[str], bool) -> str '''Turns address into RFC-2047 compliant header.''' if display or not address: return encoding.strfromlocal(address or b'') name, addr = email.utils.parseaddr(encoding.strfromlocal(address)) return _addressencode(ui, name, addr, charsets) def addrlistencode(ui, addrs, charsets=None, display=False): # type: (Any, List[bytes], List[str], bool) -> List[str] """Turns a list of addresses into a list of RFC-2047 compliant headers. A single element of input list may contain multiple addresses, but output always has one address per item""" straddrs = [] for a in addrs: assert isinstance(a, bytes), '%r unexpectedly not a bytestr' % a straddrs.append(encoding.strfromlocal(a)) if display: return [a.strip() for a in straddrs if a.strip()] result = [] for name, addr in email.utils.getaddresses(straddrs): if name or addr: r = _addressencode(ui, name, addr, charsets) result.append(r) return result def mimeencode(ui, s, charsets=None, display=False): # type: (Any, bytes, List[str], bool) -> email.message.Message """creates mime text object, encodes it if needed, and sets charset and transfer-encoding accordingly.""" cs = 'us-ascii' if not display: s, cs = _encode(ui, s, charsets) return mimetextqp(s, 'plain', cs) Generator = email.generator.BytesGenerator def parse(fp): # type: (Any) -> email.message.Message ep = email.parser.Parser() # disable the "universal newlines" mode, which isn't binary safe. # I have no idea if ascii/surrogateescape is correct, but that's # what the standard Python email parser does. fp = io.TextIOWrapper( fp, encoding='ascii', errors='surrogateescape', newline=chr(10) ) try: return ep.parse(fp) finally: fp.detach() def parsebytes(data): # type: (bytes) -> email.message.Message ep = email.parser.BytesParser() return ep.parsebytes(data) def headdecode(s): # type: (Union[email.header.Header, bytes]) -> bytes '''Decodes RFC-2047 header''' uparts = [] for part, charset in email.header.decode_header(s): if charset is not None: try: uparts.append(part.decode(charset)) continue except (UnicodeDecodeError, LookupError): pass # On Python 3, decode_header() may return either bytes or unicode # depending on whether the header has =?<charset>? or not if isinstance(part, type(u'')): uparts.append(part) continue try: uparts.append(part.decode('UTF-8')) continue except UnicodeDecodeError: pass uparts.append(part.decode('ISO-8859-1')) return encoding.unitolocal(u' '.join(uparts))