Mercurial > hg
view hgext/strip.py @ 26380:56a640b0f656
revlog: don't flush data file after every added revision
The current behavior of revlogs is to flush the data file when writing
data to it. Tracing system calls revealed that changegroup processing
incurred numerous write(2) calls for values much smaller than the
default buffer size (Python defaults to 4096, but it can be adjusted
based on detected block size at run time by CPython).
The reason we flush revlogs is so readers have all data available.
For example, the current code in revlog.py will re-open the revlog
file (instead of seeking an existing file handle) to read the text
of a revision. This happens when starting a new delta chain when
adding several revisions from changegroups, for example. Yes, this
is likely sub-optimal (we should probably be sharing file descriptors
between readers and writers to avoid the flushing and associated
overhead of re-opening files).
While flushing revlogs is necessary, it appears all callers are
diligent about flushing files before a read is performed (see
buildtext() in _addrevision()), making the flush in
_writeentry() redundant and unncessary. So, we remove it. In practice,
this means we incur a write(2) a) when the buffer is full (typically
4096 bytes) b) when a new delta chain is created rather than after
every added revision. This applies to every revlog, but by volume
it mostly impacts filelogs.
Removing the redundant flush from _writeentry() significantly
reduces the number of write(2) calls during changegroup processing on
my Linux machine. When applying a changegroup of the hg repo based on
my local repo, the total number of write(2) calls during application
of the mercurial/localrepo.py revlogs dropped from 1,320 to 217 with
this patch applied. Total I/O related system calls dropped from 1,577
to 474.
When unbundling a mozilla-central gzipped bundle (264,403 changesets
with 1,492,215 changes to 222,507 files), total write(2) calls
dropped from 1,252,881 to 827,106 and total system calls dropped from
3,601,259 to 3,178,636 - a reduction of 425,775!
While the system call reduction is significant, it appears
to have no impact on wall time on my Linux and Windows machines. Still,
fewer syscalls is fewer syscalls. Surely this can't hurt. If nothing
else, it makes examining remaining system call usage simpler and opens
the door to experimenting with the performance impact of different
buffer sizes.
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Sat, 26 Sep 2015 21:43:13 -0700 |
parents | 80c5b2666a96 |
children | 56b2bcea2529 |
line wrap: on
line source
"""strip changesets and their descendants from history This extension allows you to strip changesets and all their descendants from the repository. See the command help for details. """ from mercurial.i18n import _ from mercurial.node import nullid from mercurial.lock import release from mercurial import cmdutil, hg, scmutil, util from mercurial import repair, bookmarks, merge cmdtable = {} command = cmdutil.command(cmdtable) # Note for extension authors: ONLY specify testedwith = 'internal' for # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should # be specifying the version(s) of Mercurial they are tested with, or # leave the attribute unspecified. testedwith = 'internal' def checksubstate(repo, baserev=None): '''return list of subrepos at a different revision than substate. Abort if any subrepos have uncommitted changes.''' inclsubs = [] wctx = repo[None] if baserev: bctx = repo[baserev] else: bctx = wctx.parents()[0] for s in sorted(wctx.substate): wctx.sub(s).bailifchanged(True) if s not in bctx.substate or bctx.sub(s).dirty(): inclsubs.append(s) return inclsubs def checklocalchanges(repo, force=False, excsuffix=''): cmdutil.checkunfinished(repo) s = repo.status() if not force: if s.modified or s.added or s.removed or s.deleted: _("local changes found") # i18n tool detection raise util.Abort(_("local changes found" + excsuffix)) if checksubstate(repo): _("local changed subrepos found") # i18n tool detection raise util.Abort(_("local changed subrepos found" + excsuffix)) return s def strip(ui, repo, revs, update=True, backup=True, force=None, bookmark=None): wlock = lock = None try: wlock = repo.wlock() lock = repo.lock() if update: checklocalchanges(repo, force=force) urev, p2 = repo.changelog.parents(revs[0]) if (util.safehasattr(repo, 'mq') and p2 != nullid and p2 in [x.node for x in repo.mq.applied]): urev = p2 hg.clean(repo, urev) repo.dirstate.write() repair.strip(ui, repo, revs, backup) marks = repo._bookmarks if bookmark: if bookmark == repo._activebookmark: bookmarks.deactivate(repo) del marks[bookmark] marks.write() ui.write(_("bookmark '%s' deleted\n") % bookmark) finally: release(lock, wlock) @command("strip", [ ('r', 'rev', [], _('strip specified revision (optional, ' 'can specify revisions without this ' 'option)'), _('REV')), ('f', 'force', None, _('force removal of changesets, discard ' 'uncommitted changes (no backup)')), ('', 'no-backup', None, _('no backups')), ('', 'nobackup', None, _('no backups (DEPRECATED)')), ('n', '', None, _('ignored (DEPRECATED)')), ('k', 'keep', None, _("do not modify working directory during " "strip")), ('B', 'bookmark', '', _("remove revs only reachable from given" " bookmark"))], _('hg strip [-k] [-f] [-n] [-B bookmark] [-r] REV...')) def stripcmd(ui, repo, *revs, **opts): """strip changesets and all their descendants from the repository The strip command removes the specified changesets and all their descendants. If the working directory has uncommitted changes, the operation is aborted unless the --force flag is supplied, in which case changes will be discarded. If a parent of the working directory is stripped, then the working directory will automatically be updated to the most recent available ancestor of the stripped parent after the operation completes. Any stripped changesets are stored in ``.hg/strip-backup`` as a bundle (see :hg:`help bundle` and :hg:`help unbundle`). They can be restored by running :hg:`unbundle .hg/strip-backup/BUNDLE`, where BUNDLE is the bundle file created by the strip. Note that the local revision numbers will in general be different after the restore. Use the --no-backup option to discard the backup bundle once the operation completes. Strip is not a history-rewriting operation and can be used on changesets in the public phase. But if the stripped changesets have been pushed to a remote repository you will likely pull them again. Return 0 on success. """ backup = True if opts.get('no_backup') or opts.get('nobackup'): backup = False cl = repo.changelog revs = list(revs) + opts.get('rev') revs = set(scmutil.revrange(repo, revs)) wlock = repo.wlock() try: if opts.get('bookmark'): mark = opts.get('bookmark') marks = repo._bookmarks if mark not in marks: raise util.Abort(_("bookmark '%s' not found") % mark) # If the requested bookmark is not the only one pointing to a # a revision we have to only delete the bookmark and not strip # anything. revsets cannot detect that case. uniquebm = True for m, n in marks.iteritems(): if m != mark and n == repo[mark].node(): uniquebm = False break if uniquebm: rsrevs = repo.revs("ancestors(bookmark(%s)) - " "ancestors(head() and not bookmark(%s)) - " "ancestors(bookmark() and not bookmark(%s))", mark, mark, mark) revs.update(set(rsrevs)) if not revs: del marks[mark] marks.write() ui.write(_("bookmark '%s' deleted\n") % mark) if not revs: raise util.Abort(_('empty revision set')) descendants = set(cl.descendants(revs)) strippedrevs = revs.union(descendants) roots = revs.difference(descendants) update = False # if one of the wdir parent is stripped we'll need # to update away to an earlier revision for p in repo.dirstate.parents(): if p != nullid and cl.rev(p) in strippedrevs: update = True break rootnodes = set(cl.node(r) for r in roots) q = getattr(repo, 'mq', None) if q is not None and q.applied: # refresh queue state if we're about to strip # applied patches if cl.rev(repo.lookup('qtip')) in strippedrevs: q.applieddirty = True start = 0 end = len(q.applied) for i, statusentry in enumerate(q.applied): if statusentry.node in rootnodes: # if one of the stripped roots is an applied # patch, only part of the queue is stripped start = i break del q.applied[start:end] q.savedirty() revs = sorted(rootnodes) if update and opts.get('keep'): urev, p2 = repo.changelog.parents(revs[0]) if (util.safehasattr(repo, 'mq') and p2 != nullid and p2 in [x.node for x in repo.mq.applied]): urev = p2 uctx = repo[urev] # only reset the dirstate for files that would actually change # between the working context and uctx descendantrevs = repo.revs("%s::." % uctx.rev()) changedfiles = [] for rev in descendantrevs: # blindly reset the files, regardless of what actually changed changedfiles.extend(repo[rev].files()) # reset files that only changed in the dirstate too dirstate = repo.dirstate dirchanges = [f for f in dirstate if dirstate[f] != 'n'] changedfiles.extend(dirchanges) repo.dirstate.rebuild(urev, uctx.manifest(), changedfiles) repo.dirstate.write() # clear resolve state ms = merge.mergestate(repo) ms.reset(repo['.'].node()) update = False strip(ui, repo, revs, backup=backup, update=update, force=opts.get('force'), bookmark=opts.get('bookmark')) finally: wlock.release() return 0