discovery: slowly increase sampling size
Some pathological discovery runs can requires many roundtrip. When this happens
things can get very slow.
To make the algorithm more resilience again such pathological case. We slowly
increase the sample size with each roundtrip (+5%). This will have a negligible
impact on "normal" discovery with few roundtrips, but a large positive impact of
case with many roundtrips. Asking more question per roundtrip helps to reduce
the undecided set faster. Instead of reducing the undecided set a linear speed
(in the worst case), we reduce it as a guaranteed (small) exponential rate. The
data below show this slow ramp up in sample size:
round trip | 1 | 5 | 10 | 20 | 50 | 100 | 130 |
sample size | 200 | 254 | 321 | 517 | 2 199 | 25 123 | 108 549 |
covered nodes | 200 | 1 357 | 2 821 | 7 031 | 42 658 | 524 530 | 2 276 755 |
To be a bit more concrete, lets take a very pathological case as an example. We
are doing discovery from a copy of Mozilla-try to a more recent version of
mozilla-unified. Mozilla-unified heads are unknown to the mozilla-try repo and
there are over 1 million "missing" changesets. (the discovery is "local" to
avoid network interference)
Without this change, the discovery:
- last 1858 seconds (31 minutes),
- does 1700 round trip,
- asking about 340 000 nodes.
With this change, the discovery:
- last 218 seconds (3 minutes, 38 seconds a -88% improvement),
- does 94 round trip (-94%),
- asking about 344 211 nodes (+1%).
Of course, this is an extreme case (and 3 minutes is still slow). However this
give a good example of how this sample size increase act as a safety net
catching any bad situations.
We could image a steeper increase than 5%. For example 10% would give the
following number:
round trip | 1 | 5 | 10 | 20 | 50 | 75 | 100 |
sample size | 200 | 321 | 514 | 1 326 | 23 060 | 249 812 | 2 706 594 |
covered nodes | 200 | 1 541 | 3 690 | 12 671 | 251 871 | 2 746 254 | 29 770 966 |
In parallel, it is useful to understand these pathological cases and improve
them. However the current change provides a general purpose safety net to smooth
the impact of pathological cases.
To avoid issue with older http server, the increase in sample size only occurs
if the protocol has not limit on command argument size.
# Copyright (C) 2006 - Marco Barisione <marco@barisione.org>
#
# This is a small extension for Mercurial (https://mercurial-scm.org/)
# that removes files not known to mercurial
#
# This program was inspired by the "cvspurge" script contained in CVS
# utilities (http://www.red-bean.com/cvsutils/).
#
# For help on the usage of "hg purge" use:
# hg help purge
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, see <http://www.gnu.org/licenses/>.
'''command to delete untracked files from the working directory'''
from __future__ import absolute_import
from mercurial.i18n import _
from mercurial import (
cmdutil,
merge as mergemod,
pycompat,
registrar,
scmutil,
)
cmdtable = {}
command = registrar.command(cmdtable)
# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
# extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
# be specifying the version(s) of Mercurial they are tested with, or
# leave the attribute unspecified.
testedwith = 'ships-with-hg-core'
@command('purge|clean',
[('a', 'abort-on-err', None, _('abort if an error occurs')),
('', 'all', None, _('purge ignored files too')),
('', 'dirs', None, _('purge empty directories')),
('', 'files', None, _('purge files')),
('p', 'print', None, _('print filenames instead of deleting them')),
('0', 'print0', None, _('end filenames with NUL, for use with xargs'
' (implies -p/--print)')),
] + cmdutil.walkopts,
_('hg purge [OPTION]... [DIR]...'),
helpcategory=command.CATEGORY_MAINTENANCE)
def purge(ui, repo, *dirs, **opts):
'''removes files not tracked by Mercurial
Delete files not known to Mercurial. This is useful to test local
and uncommitted changes in an otherwise-clean source tree.
This means that purge will delete the following by default:
- Unknown files: files marked with "?" by :hg:`status`
- Empty directories: in fact Mercurial ignores directories unless
they contain files under source control management
But it will leave untouched:
- Modified and unmodified tracked files
- Ignored files (unless --all is specified)
- New files added to the repository (with :hg:`add`)
The --files and --dirs options can be used to direct purge to delete
only files, only directories, or both. If neither option is given,
both will be deleted.
If directories are given on the command line, only files in these
directories are considered.
Be careful with purge, as you could irreversibly delete some files
you forgot to add to the repository. If you only want to print the
list of files that this program would delete, use the --print
option.
'''
opts = pycompat.byteskwargs(opts)
act = not opts.get('print')
eol = '\n'
if opts.get('print0'):
eol = '\0'
act = False # --print0 implies --print
removefiles = opts.get('files')
removedirs = opts.get('dirs')
if not removefiles and not removedirs:
removefiles = True
removedirs = True
match = scmutil.match(repo[None], dirs, opts)
paths = mergemod.purge(
repo, match, ignored=opts.get('all', False),
removeemptydirs=removedirs, removefiles=removefiles,
abortonerror=opts.get('abort_on_err'),
noop=not act)
for path in paths:
if not act:
ui.write('%s%s' % (path, eol))