Mercurial > hg
view hgext/infinitepush/store.py @ 46018:8b0a3ff5ed12
statprof: separate functions and "line", assume 4 digit line numbers
Previously, the profile output looked like this (I've removed many lines that
are mostly inconsequential):
```
| 100.0% 0.02s hg: <module> line 43: dispatch.run()
| 100.0% 0.02s dispatch.py: run line 115: status = dispatch(req)
| 100.0% 0.02s dispatch.py: _runcatchfunc line 432: return _dispatch(req)
\ 50.0% 0.01s dispatch.py: _dispatch line 1228: return runcommand(
| 50.0% 0.01s dispatch.py: runcommand line 883: ret = _runcommand(ui, optio...
| 50.0% 0.01s dispatch.py: _runcommand line 1240: return cmdfunc()
| 50.0% 0.01s localrepo.py: __getitem__ line 1670: quick_access = self._quick_...
| 50.0% 0.01s localrepo.py: _quick_access_changeidline 1650: return self._quick_access_c...
| 50.0% 0.01s localrepo.py: __get__ line 179: return getattr(unfi, self.n...
| 50.0% 0.01s util.py: __get__ line 1747: result = self.func(obj)
| 50.0% 0.01s localrepo.py: _quick_access_changeid_wcline 1611: cl = self.unfiltered().chan...
| 50.0% 0.01s localrepo.py: __get__ line 110: return super(_basefilecache...
| 50.0% 0.01s util.py: __getattribute__line 245: self.__spec__.loader.exec_m...
| 50.0% 0.01s <frozen importlib._bootstrap_external>: exec_moduleline 783:
| 50.0% 0.01s <frozen importlib._bootstrap>: _call_with_frames_removedline 219:
| 50.0% 0.01s changelog.py: <module> line 376: class changelog(revlog.revl...
| 50.0% 0.01s util.py: __getattribute__line 245: self.__spec__.loader.exec_m...
| 50.0% 0.01s <frozen importlib._bootstrap_external>: exec_moduleline 779:
| 50.0% 0.01s <frozen importlib._bootstrap_external>: get_codeline 868:
| 50.0% 0.01s <frozen importlib._bootstrap_external>: path_statsline 1012:
| 50.0% 0.01s <frozen importlib._bootstrap_external>: _path_statline 87:
```
This has a few problems, though I'm only addressing some of them.
1. If the stuff before "line ###" is long, there's no separation between the
function name and the "line" string.
2. If the stuff before "line ###" is really long, there's excessive separation
between the "line" string and the line number.
3. We frequently have 4-digit line numbers, the code on the right wasn't
dynamically indented and ended up quite messy looking.
To solve these problems, I've added a ", " prefix before "line" iff it would
otherwise not have any separation such as spaces. I've added a 'max' so that we
never use a negative width (which is the cause of problem #2 above), and I've
added a default assumption of 4 digit line numbers (but again using a 'max' so
this shouldn't cause problems if we go beyond that.
With these changes, it now looks like this:
```
| 100.0% 0.02s hg: <module> line 43: dispatch.run()
| 100.0% 0.02s dispatch.py: run line 115: status = dispatch(req)
| 100.0% 0.02s dispatch.py: _runcatchfunc line 432: return _dispatch(req)
\ 50.0% 0.01s dispatch.py: _dispatch line 1228: return runcommand(
| 50.0% 0.01s dispatch.py: runcommand line 883: ret = _runcommand(ui, optio...
| 50.0% 0.01s dispatch.py: _runcommand line 1240: return cmdfunc()
| 50.0% 0.01s localrepo.py: __getitem__ line 1670: quick_access = self._quick_...
| 50.0% 0.01s localrepo.py: _quick_access_changeid, line 1650: return self._quick_access_c...
| 50.0% 0.01s localrepo.py: __get__ line 179: return getattr(unfi, self.n...
| 50.0% 0.01s util.py: __get__ line 1747: result = self.func(obj)
| 50.0% 0.01s localrepo.py: _quick_access_changeid_wc, line 1611: cl = self.unfiltered().chan...
| 50.0% 0.01s localrepo.py: __get__ line 110: return super(_basefilecache...
| 50.0% 0.01s util.py: __getattribute__, line 245: self.__spec__.loader.exec_m...
| 50.0% 0.01s <frozen importlib._bootstrap_external>: exec_module, line 783:
| 50.0% 0.01s <frozen importlib._bootstrap>: _call_with_frames_removed, line 219:
| 50.0% 0.01s changelog.py: <module> line 376: class changelog(revlog.revl...
| 50.0% 0.01s util.py: __getattribute__, line 245: self.__spec__.loader.exec_m...
| 50.0% 0.01s <frozen importlib._bootstrap_external>: exec_module, line 779:
| 50.0% 0.01s <frozen importlib._bootstrap_external>: get_code, line 868:
| 50.0% 0.01s <frozen importlib._bootstrap_external>: path_stats, line 1012:
| 50.0% 0.01s <frozen importlib._bootstrap_external>: _path_stat, line 87:
```
Differential Revision: https://phab.mercurial-scm.org/D9511
author | Kyle Lippincott <spectral@google.com> |
---|---|
date | Wed, 02 Dec 2020 12:33:51 -0800 |
parents | bfc6e75c0114 |
children | 59fa3890d40a |
line wrap: on
line source
# This software may be used and distributed according to the terms of the # GNU General Public License version 2 or any later version. # based on bundleheads extension by Gregory Szorc <gps@mozilla.com> from __future__ import absolute_import import abc import os import subprocess from mercurial.pycompat import open from mercurial import ( node, pycompat, ) from mercurial.utils import ( hashutil, procutil, ) class BundleWriteException(Exception): pass class BundleReadException(Exception): pass class abstractbundlestore(object): # pytype: disable=ignored-metaclass """Defines the interface for bundle stores. A bundle store is an entity that stores raw bundle data. It is a simple key-value store. However, the keys are chosen by the store. The keys can be any Python object understood by the corresponding bundle index (see ``abstractbundleindex`` below). """ __metaclass__ = abc.ABCMeta @abc.abstractmethod def write(self, data): """Write bundle data to the store. This function receives the raw data to be written as a str. Throws BundleWriteException The key of the written data MUST be returned. """ @abc.abstractmethod def read(self, key): """Obtain bundle data for a key. Returns None if the bundle isn't known. Throws BundleReadException The returned object should be a file object supporting read() and close(). """ class filebundlestore(object): """bundle store in filesystem meant for storing bundles somewhere on disk and on network filesystems """ def __init__(self, ui, repo): self.ui = ui self.repo = repo self.storepath = ui.configpath(b'scratchbranch', b'storepath') if not self.storepath: self.storepath = self.repo.vfs.join( b"scratchbranches", b"filebundlestore" ) if not os.path.exists(self.storepath): os.makedirs(self.storepath) def _dirpath(self, hashvalue): """First two bytes of the hash are the name of the upper level directory, next two bytes are the name of the next level directory""" return os.path.join(self.storepath, hashvalue[0:2], hashvalue[2:4]) def _filepath(self, filename): return os.path.join(self._dirpath(filename), filename) def write(self, data): filename = node.hex(hashutil.sha1(data).digest()) dirpath = self._dirpath(filename) if not os.path.exists(dirpath): os.makedirs(dirpath) with open(self._filepath(filename), b'wb') as f: f.write(data) return filename def read(self, key): try: with open(self._filepath(key), b'rb') as f: return f.read() except IOError: return None def format_placeholders_args(args, filename=None, handle=None): """Formats `args` with Infinitepush replacements. Hack to get `str.format()`-ed strings working in a BC way with bytes. """ formatted_args = [] for arg in args: if filename and arg == b'{filename}': formatted_args.append(filename) elif handle and arg == b'{handle}': formatted_args.append(handle) else: formatted_args.append(arg) return formatted_args class externalbundlestore(abstractbundlestore): def __init__(self, put_binary, put_args, get_binary, get_args): """ `put_binary` - path to binary file which uploads bundle to external storage and prints key to stdout `put_args` - format string with additional args to `put_binary` {filename} replacement field can be used. `get_binary` - path to binary file which accepts filename and key (in that order), downloads bundle from store and saves it to file `get_args` - format string with additional args to `get_binary`. {filename} and {handle} replacement field can be used. """ self.put_args = put_args self.get_args = get_args self.put_binary = put_binary self.get_binary = get_binary def _call_binary(self, args): p = subprocess.Popen( pycompat.rapply(procutil.tonativestr, args), stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True, ) stdout, stderr = p.communicate() returncode = p.returncode return returncode, stdout, stderr def write(self, data): # Won't work on windows because you can't open file second time without # closing it # TODO: rewrite without str.format() and replace NamedTemporaryFile() # with pycompat.namedtempfile() with pycompat.namedtempfile() as temp: temp.write(data) temp.flush() temp.seek(0) formatted_args = format_placeholders_args( self.put_args, filename=temp.name ) returncode, stdout, stderr = self._call_binary( [self.put_binary] + formatted_args ) if returncode != 0: raise BundleWriteException( b'Failed to upload to external store: %s' % stderr ) stdout_lines = stdout.splitlines() if len(stdout_lines) == 1: return stdout_lines[0] else: raise BundleWriteException( b'Bad output from %s: %s' % (self.put_binary, stdout) ) def read(self, handle): # Won't work on windows because you can't open file second time without # closing it with pycompat.namedtempfile() as temp: formatted_args = format_placeholders_args( self.get_args, filename=temp.name, handle=handle ) returncode, stdout, stderr = self._call_binary( [self.get_binary] + formatted_args ) if returncode != 0: raise BundleReadException( b'Failed to download from external store: %s' % stderr ) return temp.read()