view hgext/censor.py @ 35777:0c0689a7565e

subrepo: handle 'C:' style paths on the command line (issue5770)

If you think 'C:' and 'C:\' are equivalent paths, see the inline comment before
proceeding.

The problem here was that several commands that take a URL argument (incoming,
outgoing, pull, and push) will use that value to set 'repo._subtoppath' on the
repository object after command specific manipulation of it, but before
converting it to an absolute path.  When an operation is performed on a relative
subrepo, subrepo._abssource() will posixpath.join() this value with the relative
subrepo path.  That adds a '/' after the drive letter, changing how it is
evaluated by abspath()/realpath() in vfsmod.vfs(..., realpath=True) as the
subrepo is instantiated.

I initially tried sanitizing the path in url.localpath(), because url.isabs()
only checks that it starts with a drive letter.  By the sample behavior, this is
clearly not an absolute path.  (Though the comment in isabs() is weasely- this
style path can't be joined either.)  But not everything funnels through there,
and it required explicitly calling localpath() in hg.parseurl() and assigning to
url.path to fix.  But then tests failed with urls like 'a#0'.

Next up was sanitizing the path in the url constructor.  That caused doctest
failures, because there are drive letter tests, so those got expanded in system
specific ways.  Yuya correctly pointed out that util.url is a parser, and
shouldn't be substituting the path too.

Rather than fixing every command call site, just convert it in the common
subrepo location.  I don't see any sanitizing on the path config options, so I
fixed those too.  Note that while the behavior is fixed here, there are still
places where 'comparing with C:' gets printed out, and that's not great for
debugging purposes.  (Specifically I saw it in `hg incoming -B C:`, without
subrepos.)  While clone will write out an absolute default path, I wonder what
would happen if a user edited that path to be 'C:'.  (I don't think supporting
relative paths in .hgrc is a sane thing to do, but while we're poking holes in
things...)

Since this is such an oddball case, it still leaks through in places, and there
seems to be a lot of duplicate url parsing, maybe the url parsing should be
moved to dispatch, and provide the command with a url object?  Then we could
convert this to an absolute path once, and not have to worry about it in the
rest of the code.

I also checked '--cwd C:' on the command line, and it was previously working
because os.chdir() will DTRT.

Finally, one other note from the url.localpath() experimenting.  I don't see any
cases where 'self._hostport' can hold a drive letter.  So I'm wondering if that
is wrong/old code.

# Copyright (C) 2015 - Mike Edgar <[e-mail]>
#
# This extension enables removal of file content at a given revision,
# rewriting the data/metadata of successive revisions to preserve revision log
# integrity.

"""erase file content at a given revision

The censor command instructs Mercurial to erase all content of a file at a given
revision *without updating the changeset hash.* This allows existing history to
remain valid while preventing future clones/pulls from receiving the erased
data.

Typical uses for censor are due to security or legal requirements, including::

 * Passwords, private keys, cryptographic material
 * Licensed data/code/libraries for which the license has expired
 * Personally Identifiable Information or other private data

Censored nodes can interrupt mercurial's typical operation whenever the excised
data needs to be materialized. Some commands, like ``hg cat``/``hg revert``,
simply fail when asked to produce censored data. Others, like ``hg verify`` and
``hg update``, must be capable of tolerating censored data to continue to
function in a meaningful way. Such commands only tolerate censored file
revisions if they are allowed by the "censor.policy=ignore" config option.
"""

from __future__ import absolute_import

from mercurial.i18n import _
from mercurial.node import short

from mercurial import (
    error,
    filelog,
    lock as lockmod,
    registrar,
    revlog,
    scmutil,
    util,
)

cmdtable = {}
command = registrar.command(cmdtable)
# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
# extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
# be specifying the version(s) of Mercurial they are tested with, or
# leave the attribute unspecified.
testedwith = 'ships-with-hg-core'

@command('censor',
    [('r', 'rev', '', _('censor file from specified revision'), _('REV')),
     ('t', 'tombstone', '', _('replacement tombstone data'), _('TEXT'))],
    _('-r REV [-t TEXT] [FILE]'))
def censor(ui, repo, path, rev='', tombstone='', **opts):
    wlock = lock = None
    try:
        wlock = repo.wlock()
        lock = repo.lock()
        return _docensor(ui, repo, path, rev, tombstone, **opts)
    finally:
        lockmod.release(lock, wlock)

def _docensor(ui, repo, path, rev='', tombstone='', **opts):
    if not path:
        raise error.Abort(_('must specify file path to censor'))
    if not rev:
        raise error.Abort(_('must specify revision to censor'))

    wctx = repo[None]

    m = scmutil.match(wctx, (path,))
    if m.anypats() or len(m.files()) != 1:
        raise error.Abort(_('can only specify an explicit filename'))
    path = m.files()[0]
    flog = repo.file(path)
    if not len(flog):
        raise error.Abort(_('cannot censor file with no history'))

    rev = scmutil.revsingle(repo, rev, rev).rev()
    try:
        ctx = repo[rev]
    except KeyError:
        raise error.Abort(_('invalid revision identifier %s') % rev)

    try:
        fctx = ctx.filectx(path)
    except error.LookupError:
        raise error.Abort(_('file does not exist at revision %s') % rev)

    fnode = fctx.filenode()
    headctxs = [repo[c] for c in repo.heads()]
    heads = [c for c in headctxs if path in c and c.filenode(path) == fnode]
    if heads:
        headlist = ', '.join([short(c.node()) for c in heads])
        raise error.Abort(_('cannot censor file in heads (%s)') % headlist,
            hint=_('clean/delete and commit first'))

    wp = wctx.parents()
    if ctx.node() in [p.node() for p in wp]:
        raise error.Abort(_('cannot censor working directory'),
            hint=_('clean/delete/update first'))

    flogv = flog.version & 0xFFFF
    if flogv != revlog.REVLOGV1:
        raise error.Abort(
            _('censor does not support revlog version %d') % (flogv,))

    tombstone = filelog.packmeta({"censored": tombstone}, "")

    crev = fctx.filerev()

    if len(tombstone) > flog.rawsize(crev):
        raise error.Abort(_(
            'censor tombstone must be no longer than censored data'))

    # Using two files instead of one makes it easy to rewrite entry-by-entry
    idxread = repo.svfs(flog.indexfile, 'r')
    idxwrite = repo.svfs(flog.indexfile, 'wb', atomictemp=True)
    if flog.version & revlog.FLAG_INLINE_DATA:
        dataread, datawrite = idxread, idxwrite
    else:
        dataread = repo.svfs(flog.datafile, 'r')
        datawrite = repo.svfs(flog.datafile, 'wb', atomictemp=True)

    # Copy all revlog data up to the entry to be censored.
    rio = revlog.revlogio()
    offset = flog.start(crev)

    for chunk in util.filechunkiter(idxread, limit=crev * rio.size):
        idxwrite.write(chunk)
    for chunk in util.filechunkiter(dataread, limit=offset):
        datawrite.write(chunk)

    def rewriteindex(r, newoffs, newdata=None):
        """Rewrite the index entry with a new data offset and optional new data.

        The newdata argument, if given, is a tuple of three positive integers:
        (new compressed, new uncompressed, added flag bits).
        """
        offlags, comp, uncomp, base, link, p1, p2, nodeid = flog.index[r]
        flags = revlog.gettype(offlags)
        if newdata:
            comp, uncomp, nflags = newdata
            flags |= nflags
        offlags = revlog.offset_type(newoffs, flags)
        e = (offlags, comp, uncomp, r, link, p1, p2, nodeid)
        idxwrite.write(rio.packentry(e, None, flog.version, r))
        idxread.seek(rio.size, 1)

    def rewrite(r, offs, data, nflags=revlog.REVIDX_DEFAULT_FLAGS):
        """Write the given full text to the filelog with the given data offset.

        Returns:
            The integer number of data bytes written, for tracking data offsets.
        """
        flag, compdata = flog.compress(data)
        newcomp = len(flag) + len(compdata)
        rewriteindex(r, offs, (newcomp, len(data), nflags))
        datawrite.write(flag)
        datawrite.write(compdata)
        dataread.seek(flog.length(r), 1)
        return newcomp

    # Rewrite censored revlog entry with (padded) tombstone data.
    pad = ' ' * (flog.rawsize(crev) - len(tombstone))
    offset += rewrite(crev, offset, tombstone + pad, revlog.REVIDX_ISCENSORED)

    # Rewrite all following filelog revisions fixing up offsets and deltas.
    for srev in xrange(crev + 1, len(flog)):
        if crev in flog.parentrevs(srev):
            # Immediate children of censored node must be re-added as fulltext.
            try:
                revdata = flog.revision(srev)
            except error.CensoredNodeError as e:
                revdata = e.tombstone
            dlen = rewrite(srev, offset, revdata)
        else:
            # Copy any other revision data verbatim after fixing up the offset.
            rewriteindex(srev, offset)
            dlen = flog.length(srev)
            for chunk in util.filechunkiter(dataread, limit=dlen):
                datawrite.write(chunk)
        offset += dlen

    idxread.close()
    idxwrite.close()
    if dataread is not idxread:
        dataread.close()
        datawrite.close()