commit: improve the files field of changelog for merges
Currently, the files list of merge commits repeats all the deletions
(either actual deletions, or files that got renamed) that happened
between base and p2 of the merge. If p2 is the main branch, the list
can easily be much bigger than the change being merged.
This results in various problems worth improving:
- changelog is bigger than necessary
- `hg log directory` lists many unrelated merge commits, and `hg log
-v -r commit` frequently fills multiple screens worth of files
- it possibly slows down adjustlinkrev, by forcing it to read more
manifests, and that function can certainly be a bottleneck
- the server side of pulls can waste a lot of time simply opening the
filelogs for pointless files (the constant factors for opening even
a tiny filelog is apparently pretty bad)
So stop listing such files as described in the code. Impacted merge
commits and their descendants get a different hash than they would
have without this. This doesn't seem problematic, except for
convert. The previous commit helped with that in the hg->hg case (but
if you do svn->hg twice from scratch, hashes can still change).
The rest of the description is numbers. I don't have much to report,
because recreating the files list of existing repositories is not
easy:
- debugupgradeformat and bundle/unbundle don't recreate the list
- export/import tends to choke quickly applying patches or on
description that contain diffs,
- merge commits from the convert extension don't have the right files
list for reasons orthogonal to the current commit
- replaying the merge with hg update/hg merge/hg revert --all/hg
commit can end up failing in hg revert
- I wasn't sure that using debugsetparents + debugrebuilddirstate
would really build the right thing
I measured commit time before and after this change, in a case with no
files filtered out, several files filtered out (no difference) and 5k
files filtered out (+1% time).
Recreating the 100 more recent merges in a private repo, the
concatenated uncompressed files lists goes from 1.12MB to
0.52MB. Excluding 3 merges that are not representative, then the size
goes from 570k to 15k.
I converted part of mozilla-central, and observed file list shrinking
quite a bit too, starting at the very first merge,
733641d9feaf, going
from 550 files to 10 files (although they have relatively few merges,
so they probably wouldn't care).
Differential Revision: https://phab.mercurial-scm.org/D6613
# automv.py
#
# Copyright 2013-2016 Facebook, Inc.
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
"""check for unrecorded moves at commit time (EXPERIMENTAL)
This extension checks at commit/amend time if any of the committed files
comes from an unrecorded mv.
The threshold at which a file is considered a move can be set with the
``automv.similarity`` config option. This option takes a percentage between 0
(disabled) and 100 (files must be identical), the default is 95.
"""
# Using 95 as a default similarity is based on an analysis of the mercurial
# repositories of the cpython, mozilla-central & mercurial repositories, as
# well as 2 very large facebook repositories. At 95 50% of all potential
# missed moves would be caught, as well as correspond with 87% of all
# explicitly marked moves. Together, 80% of moved files are 95% similar or
# more.
#
# See http://markmail.org/thread/5pxnljesvufvom57 for context.
from __future__ import absolute_import
from mercurial.i18n import _
from mercurial import (
commands,
copies,
error,
extensions,
pycompat,
registrar,
scmutil,
similar
)
configtable = {}
configitem = registrar.configitem(configtable)
configitem('automv', 'similarity',
default=95,
)
def extsetup(ui):
entry = extensions.wrapcommand(
commands.table, 'commit', mvcheck)
entry[1].append(
('', 'no-automv', None,
_('disable automatic file move detection')))
def mvcheck(orig, ui, repo, *pats, **opts):
"""Hook to check for moves at commit time"""
opts = pycompat.byteskwargs(opts)
renames = None
disabled = opts.pop('no_automv', False)
if not disabled:
threshold = ui.configint('automv', 'similarity')
if not 0 <= threshold <= 100:
raise error.Abort(_('automv.similarity must be between 0 and 100'))
if threshold > 0:
match = scmutil.match(repo[None], pats, opts)
added, removed = _interestingfiles(repo, match)
uipathfn = scmutil.getuipathfn(repo, legacyrelativevalue=True)
renames = _findrenames(repo, uipathfn, added, removed,
threshold / 100.0)
with repo.wlock():
if renames is not None:
scmutil._markchanges(repo, (), (), renames)
return orig(ui, repo, *pats, **pycompat.strkwargs(opts))
def _interestingfiles(repo, matcher):
"""Find what files were added or removed in this commit.
Returns a tuple of two lists: (added, removed). Only files not *already*
marked as moved are included in the added list.
"""
stat = repo.status(match=matcher)
added = stat.added
removed = stat.removed
copy = copies.pathcopies(repo['.'], repo[None], matcher)
# remove the copy files for which we already have copy info
added = [f for f in added if f not in copy]
return added, removed
def _findrenames(repo, uipathfn, added, removed, similarity):
"""Find what files in added are really moved files.
Any file named in removed that is at least similarity% similar to a file
in added is seen as a rename.
"""
renames = {}
if similarity > 0:
for src, dst, score in similar.findrenames(
repo, added, removed, similarity):
if repo.ui.verbose:
repo.ui.status(
_('detected move of %s as %s (%d%% similar)\n') % (
uipathfn(src), uipathfn(dst), score * 100))
renames[dst] = src
if renames:
repo.ui.status(_('detected move of %d files\n') % len(renames))
return renames