Mercurial > hg
view mercurial/i18n.py @ 38732:be4984261611
merge: mark file gets as not thread safe (issue5933)
In default installs, this has the effect of disabling the thread-based
worker on Windows when manifesting files in the working directory. My
measurements have shown that with revlog-based repositories, Mercurial
spends a lot of CPU time in revlog code resolving file data. This ends
up incurring a lot of context switching across threads and slows down
`hg update` operations when going from an empty working directory to
the tip of the repo.
On mozilla-unified (246,351 files) on an i7-6700K (4+4 CPUs):
before: 487s wall
after: 360s wall (equivalent to worker.enabled=false)
cpus=2: 379s wall
Even with only 2 threads, the thread pool is still slower.
The introduction of the thread-based worker (02b36e860e0b) states that
it resulted in a "~50%" speedup for `hg sparse --enable-profile` and
`hg sparse --disable-profile`. This disagrees with my measurement
above. I theorize a few reasons for this:
1) Removal of files from the working directory is I/O - not CPU - bound
and should benefit from a thread pool (unless I/O is insanely fast
and the GIL release is near instantaneous). So tests like `hg sparse
--enable-profile` may exercise deletion throughput and aren't good
benchmarks for worker tasks that are CPU heavy.
2) The patch was authored by someone at Facebook. The results were
likely measured against a repository using remotefilelog. And I
believe that revision retrieval during working directory updates with
remotefilelog will often use a remote store, thus being I/O and not
CPU bound. This probably resulted in an overstated performance gain.
Since there appears to be a need to enable the thread-based worker with
some stores, I've made the flagging of file gets as thread safe
configurable. I've made it experimental because I don't want to formalize
a boolean flag for this option and because this attribute is best
captured against the store implementation. But we don't have a proper
store API for this yet. I'd rather cross this bridge later.
It is possible there are revlog-based repositories that do benefit from
a thread-based worker. I didn't do very comprehensive testing. If there
are, we may want to devise a more proper algorithm for whether to use
the thread-based worker, including possibly config options to limit the
number of threads to use. But until I see evidence that justifies
complexity, simplicity wins.
Differential Revision: https://phab.mercurial-scm.org/D3963
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Wed, 18 Jul 2018 09:49:34 -0700 |
parents | 79dd61a4554f |
children | dd83aafdb64a |
line wrap: on
line source
# i18n.py - internationalization support for mercurial # # Copyright 2005, 2006 Matt Mackall <mpm@selenic.com> # # This software may be used and distributed according to the terms of the # GNU General Public License version 2 or any later version. from __future__ import absolute_import import gettext as gettextmod import locale import os import sys from . import ( encoding, pycompat, ) # modelled after templater.templatepath: if getattr(sys, 'frozen', None) is not None: module = pycompat.sysexecutable else: module = pycompat.fsencode(__file__) _languages = None if (pycompat.iswindows and 'LANGUAGE' not in encoding.environ and 'LC_ALL' not in encoding.environ and 'LC_MESSAGES' not in encoding.environ and 'LANG' not in encoding.environ): # Try to detect UI language by "User Interface Language Management" API # if no locale variables are set. Note that locale.getdefaultlocale() # uses GetLocaleInfo(), which may be different from UI language. # (See http://msdn.microsoft.com/en-us/library/dd374098(v=VS.85).aspx ) try: import ctypes langid = ctypes.windll.kernel32.GetUserDefaultUILanguage() _languages = [locale.windows_locale[langid]] except (ImportError, AttributeError, KeyError): # ctypes not found or unknown langid pass _ugettext = None def setdatapath(datapath): datapath = pycompat.fsdecode(datapath) localedir = os.path.join(datapath, r'locale') t = gettextmod.translation(r'hg', localedir, _languages, fallback=True) global _ugettext try: _ugettext = t.ugettext except AttributeError: _ugettext = t.gettext _msgcache = {} # encoding: {message: translation} def gettext(message): """Translate message. The message is looked up in the catalog to get a Unicode string, which is encoded in the local encoding before being returned. Important: message is restricted to characters in the encoding given by sys.getdefaultencoding() which is most likely 'ascii'. """ # If message is None, t.ugettext will return u'None' as the # translation whereas our callers expect us to return None. if message is None or not _ugettext: return message cache = _msgcache.setdefault(encoding.encoding, {}) if message not in cache: if type(message) is pycompat.unicode: # goofy unicode docstrings in test paragraphs = message.split(u'\n\n') else: paragraphs = [p.decode("ascii") for p in message.split('\n\n')] # Be careful not to translate the empty string -- it holds the # meta data of the .po file. u = u'\n\n'.join([p and _ugettext(p) or u'' for p in paragraphs]) try: # encoding.tolocal cannot be used since it will first try to # decode the Unicode string. Calling u.decode(enc) really # means u.encode(sys.getdefaultencoding()).decode(enc). Since # the Python encoding defaults to 'ascii', this fails if the # translated string use non-ASCII characters. encodingstr = pycompat.sysstr(encoding.encoding) cache[message] = u.encode(encodingstr, "replace") except LookupError: # An unknown encoding results in a LookupError. cache[message] = message return cache[message] def _plain(): if ('HGPLAIN' not in encoding.environ and 'HGPLAINEXCEPT' not in encoding.environ): return False exceptions = encoding.environ.get('HGPLAINEXCEPT', '').strip().split(',') return 'i18n' not in exceptions if _plain(): _ = lambda message: message else: _ = gettext