Mercurial > hg
view mercurial/pushkey.py @ 27895:2d6a89e79b48
scmutil: support background file closing
Closing files that have been appended to is relatively slow on
Windows/NTFS. This makes several Mercurial operations slower on
Windows.
The workaround to this issue is conceptually simple: use multiple
threads for I/O. Unfortunately, Python doesn't scale well to multiple
threads because of the GIL. And, refactoring our code to use threads
everywhere would be a huge undertaking. So, we decide to tackle this
problem by starting small: establishing a thread pool for closing
files.
This patch establishes a mechanism for closing file handles on separate
threads. The coordinator object is basically a queue of file handles to
operate on and a thread pool consuming from the queue.
When files are opened through the VFS layer, the caller can specify
that delay closing is allowed.
A proxy class for file handles has been added. We must use a proxy
because it isn't possible to modify __class__ on built-in types. This
adds some overhead. But as future patches will show, this overhead
is cancelled out by the benefit of closing file handles on background
threads.
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Thu, 14 Jan 2016 13:34:59 -0800 |
parents | 7b200566e474 |
children | 57875cf423c9 |
line wrap: on
line source
# pushkey.py - dispatching for pushing and pulling keys # # Copyright 2010 Matt Mackall <mpm@selenic.com> # # This software may be used and distributed according to the terms of the # GNU General Public License version 2 or any later version. from __future__ import absolute_import from . import ( bookmarks, encoding, obsolete, phases, ) def _nslist(repo): n = {} for k in _namespaces: n[k] = "" if not obsolete.isenabled(repo, obsolete.exchangeopt): n.pop('obsolete') return n _namespaces = {"namespaces": (lambda *x: False, _nslist), "bookmarks": (bookmarks.pushbookmark, bookmarks.listbookmarks), "phases": (phases.pushphase, phases.listphases), "obsolete": (obsolete.pushmarker, obsolete.listmarkers), } def register(namespace, pushkey, listkeys): _namespaces[namespace] = (pushkey, listkeys) def _get(namespace): return _namespaces.get(namespace, (lambda *x: False, lambda *x: {})) def push(repo, namespace, key, old, new): '''should succeed iff value was old''' pk = _get(namespace)[0] return pk(repo, key, old, new) def list(repo, namespace): '''return a dict''' lk = _get(namespace)[1] return lk(repo) encode = encoding.fromlocal decode = encoding.tolocal def encodekeys(keys): """encode the content of a pushkey namespace for exchange over the wire""" return '\n'.join(['%s\t%s' % (encode(k), encode(v)) for k, v in keys]) def decodekeys(data): """decode the content of a pushkey namespace from exchange over the wire""" result = {} for l in data.splitlines(): k, v = l.split('\t') result[decode(k)] = decode(v) return result