hgwebdir: avoid systematic full garbage collection
Forcing a systematic full garbage collection upon each request
can serioulsy harm performance. This is reported as
https://bz.mercurial-scm.org/show_bug.cgi?id=6075
With this change we're performing the full collection according
to a new setting, `experimental.web.full-garbage-collection-rate`.
The default value is 1, which doesn't change the behavior and will
allow us to test on real use cases. If the value is 0, no full garbage
collection occurs.
Regardless of the value of the setting, a partial garbage collection
still occurs upon each request (not attempting to collect objects from
the oldest generation). This should be enough to take care of
reference cycles that have been created by the last request
(assessment of this requires changing the setting, not to be 1).
In my experience chasing memory leaks in Mercurial servers,
the full collection never reclaimed any memory, but this is with
Python 3 and biased towards small repositories.
On the other hand, as explained in the Python developer docs [1],
frequent full collections are very harmful in terms of performance if
lots of objects survive the collection, and hence stay in the
oldest generation. Note that `gc.collect()` is indeed trying to
collect the oldest generation [2]. This happens usually in two cases:
- unwanted lingering objects (i.e., an actual memory leak that
the GC cannot do anything about). Sadly, we have lots of those
these days.
- desireable long-term objects, typically in caches (not inner caches
carried by repositories, which should be collected with them). This
is a subject of interest for the Heptapod project.
In short, the flat rate that this change still permits is
probably a bad idea in most cases, and the default value can
be tweaked later on (or even be set to 0) according to experiments
in the wild.
The test is inspired from test-hgwebdir-paths.py
[1] https://devguide.python.org/garbage_collector/#collecting-the-oldest-generation
[2] https://docs.python.org/3/library/gc.html#gc.collect
Differential Revision: https://phab.mercurial-scm.org/D11204
#!/bin/sh
#
# Use this script to generate move.svndump
#
mkdir temp
cd temp
mkdir project-orig
cd project-orig
mkdir trunk
echo a > trunk/a
mkdir trunk/d1
mkdir trunk/d2
echo b > trunk/d1/b
echo c > trunk/d1/c
echo d > trunk/d2/d
cd ..
svnadmin create svn-repo
svnurl=file://`pwd`/svn-repo
svn import project-orig $svnurl -m "init projA"
svn co $svnurl project
cd project
# Build a module renaming chain which used to confuse the converter.
# Update svn repository
echo a >> trunk/a
echo c >> trunk/d1/c
svn ci -m commitbeforemove
svn mv $svnurl/trunk $svnurl/subproject -m movedtrunk
svn up
mkdir subproject/trunk
svn add subproject/trunk
svn ci -m createtrunk
mkdir subproject/branches
svn add subproject/branches
svn ci -m createbranches
svn mv $svnurl/subproject/d1 $svnurl/subproject/trunk/d1 -m moved1
svn mv $svnurl/subproject/d2 $svnurl/subproject/trunk/d2 -m moved2
svn up
echo b >> subproject/trunk/d1/b
svn rm subproject/trunk/d2
svn ci -m "changeb and rm d2"
svn mv $svnurl/subproject/trunk/d1 $svnurl/subproject/branches/d1 -m moved1again
if svn help copy | grep 'SRC\[@REV\]' > /dev/null 2>&1; then
# SVN >= 1.5 replaced the -r REV syntax with @REV
# Copy a file from a past revision
svn copy $svnurl/subproject/trunk/d2/d@7 $svnurl/subproject/trunk -m copyfilefrompast
# Copy a directory from a past revision
svn copy $svnurl/subproject/trunk/d2@7 $svnurl/subproject/trunk -m copydirfrompast
else
# Copy a file from a past revision
svn copy -r 7 $svnurl/subproject/trunk/d2/d $svnurl/subproject/trunk -m copyfilefrompast
# Copy a directory from a past revision
svn copy -r 7 $svnurl/subproject/trunk/d2 $svnurl/subproject/trunk -m copydirfrompast
fi
# Copy a directory while removing a subdirectory
svn up
mkdir -p subproject/trunk/d3/d31
echo e > subproject/trunk/d3/d31/e
echo f > subproject/trunk/d3/f
svn add subproject/trunk/d3
svn ci -m "add d3"
svn copy subproject/trunk/d3 subproject/trunk/d4
svn rm subproject/trunk/d3/d31
svn ci -m "copy dir and remove subdir"
# Test directory moves
svn up
mkdir -p subproject/trunk/d4old
echo g > subproject/trunk/d4old/g
svn add subproject/trunk/d4old
svn ci -m "add d4old"
svn mv subproject/trunk/d4old subproject/trunk/d4new
svn ci -m "rename d4old into d4new"
cd ..
svnadmin dump svn-repo > ../move.svndump