hgwebdir: avoid systematic full garbage collection
Forcing a systematic full garbage collection upon each request
can serioulsy harm performance. This is reported as
https://bz.mercurial-scm.org/show_bug.cgi?id=6075
With this change we're performing the full collection according
to a new setting, `experimental.web.full-garbage-collection-rate`.
The default value is 1, which doesn't change the behavior and will
allow us to test on real use cases. If the value is 0, no full garbage
collection occurs.
Regardless of the value of the setting, a partial garbage collection
still occurs upon each request (not attempting to collect objects from
the oldest generation). This should be enough to take care of
reference cycles that have been created by the last request
(assessment of this requires changing the setting, not to be 1).
In my experience chasing memory leaks in Mercurial servers,
the full collection never reclaimed any memory, but this is with
Python 3 and biased towards small repositories.
On the other hand, as explained in the Python developer docs [1],
frequent full collections are very harmful in terms of performance if
lots of objects survive the collection, and hence stay in the
oldest generation. Note that `gc.collect()` is indeed trying to
collect the oldest generation [2]. This happens usually in two cases:
- unwanted lingering objects (i.e., an actual memory leak that
the GC cannot do anything about). Sadly, we have lots of those
these days.
- desireable long-term objects, typically in caches (not inner caches
carried by repositories, which should be collected with them). This
is a subject of interest for the Heptapod project.
In short, the flat rate that this change still permits is
probably a bad idea in most cases, and the default value can
be tweaked later on (or even be set to 0) according to experiments
in the wild.
The test is inspired from test-hgwebdir-paths.py
[1] https://devguide.python.org/garbage_collector/#collecting-the-oldest-generation
[2] https://docs.python.org/3/library/gc.html#gc.collect
Differential Revision: https://phab.mercurial-scm.org/D11204
SVN-fs-dump-format-version: 2
UUID: 644ede6c-2b81-4367-9dc8-d786514f2cde
Revision-number: 0
Prop-content-length: 56
Content-length: 56
K 8
svn:date
V 27
2010-05-19T20:16:07.429098Z
PROPS-END
Revision-number: 1
Prop-content-length: 112
Content-length: 112
K 7
svn:log
V 10
init projA
K 10
svn:author
V 7
pmezard
K 8
svn:date
V 27
2010-05-19T20:16:07.461283Z
PROPS-END
Node-path: branches
Node-kind: dir
Node-action: add
Prop-content-length: 10
Content-length: 10
PROPS-END
Node-path: trunk
Node-kind: dir
Node-action: add
Prop-content-length: 10
Content-length: 10
PROPS-END
Revision-number: 2
Prop-content-length: 106
Content-length: 106
K 7
svn:log
V 5
hello
K 10
svn:author
V 7
pmezard
K 8
svn:date
V 27
2010-05-19T20:16:08.121436Z
PROPS-END
Node-path: branches/notinbranch
Node-kind: file
Node-action: add
Prop-content-length: 10
Text-content-length: 2
Text-content-md5: e29311f6f1bf1af907f9ef9f44b8328b
Text-content-sha1: e983f374794de9c64e3d1c1de1d490c0756eeeff
Content-length: 12
PROPS-END
d
Node-path: trunk/a
Node-kind: file
Node-action: add
Prop-content-length: 10
Text-content-length: 2
Text-content-md5: 60b725f10c9c85c70d97880dfe8191b3
Text-content-sha1: 3f786850e387550fdab836ed7e6dc881de23001b
Content-length: 12
PROPS-END
a
Node-path: trunk/b
Node-kind: file
Node-action: add
Prop-content-length: 10
Text-content-length: 2
Text-content-md5: 3b5d5c3712955042212316173ccf37be
Text-content-sha1: 89e6c98d92887913cadf06b2adb97f26cde4849b
Content-length: 12
PROPS-END
b
Node-path: trunk/c
Node-kind: file
Node-action: add
Prop-content-length: 10
Text-content-length: 2
Text-content-md5: 2cd6ee2c70b0bde53fbe6cac3c8b8bb1
Text-content-sha1: 2b66fd261ee5c6cfc8de7fa466bab600bcfe4f69
Content-length: 12
PROPS-END
c
Node-path: trunk/dir
Node-kind: dir
Node-action: add
Prop-content-length: 10
Content-length: 10
PROPS-END
Node-path: trunk/dir/e
Node-kind: file
Node-action: add
Prop-content-length: 10
Text-content-length: 2
Text-content-md5: 9ffbf43126e33be52cd2bf7e01d627f9
Text-content-sha1: 094e3afb2fe8dfe82f63731cdcd3b999f4856cff
Content-length: 12
PROPS-END
e
Revision-number: 3
Prop-content-length: 132
Content-length: 132
K 7
svn:log
V 30
branch trunk, remove c and dir
K 10
svn:author
V 7
pmezard
K 8
svn:date
V 27
2010-05-19T20:16:11.113124Z
PROPS-END
Node-path: branches/old
Node-kind: dir
Node-action: add
Node-copyfrom-rev: 2
Node-copyfrom-path: trunk
Node-path: branches/old/dir
Node-action: delete
Node-path: branches/old/c
Node-action: delete
Revision-number: 4
Prop-content-length: 109
Content-length: 109
K 7
svn:log
V 8
change a
K 10
svn:author
V 7
pmezard
K 8
svn:date
V 27
2010-05-19T20:16:13.060877Z
PROPS-END
Node-path: trunk/a
Node-kind: file
Node-action: change
Text-content-length: 4
Text-content-md5: 0d227f1abf8c2932d342e9b99cc957eb
Text-content-sha1: d7c8127a20a396cff08af086a1c695b0636f0c29
Content-length: 4
a
a
Revision-number: 5
Prop-content-length: 109
Content-length: 109
K 7
svn:log
V 8
change b
K 10
svn:author
V 7
pmezard
K 8
svn:date
V 27
2010-05-19T20:16:14.066212Z
PROPS-END
Node-path: branches/old/b
Node-kind: file
Node-action: change
Text-content-length: 4
Text-content-md5: 06ac26ed8b614fc0b141e4542aa067c2
Text-content-sha1: f6980469e74f7125178e88ec571e06fe6ce86e95
Content-length: 4
b
b
Revision-number: 6
Prop-content-length: 119
Content-length: 119
K 7
svn:log
V 17
move and update c
K 10
svn:author
V 7
pmezard
K 8
svn:date
V 27
2010-05-19T20:16:16.069449Z
PROPS-END
Node-path: branches/old/c
Node-kind: file
Node-action: add
Node-copyfrom-rev: 3
Node-copyfrom-path: trunk/b
Text-copy-source-md5: 3b5d5c3712955042212316173ccf37be
Text-copy-source-sha1: 89e6c98d92887913cadf06b2adb97f26cde4849b
Text-content-length: 4
Text-content-md5: 33cb6785d50937d8d307ebb66d6259a7
Text-content-sha1: 7a6478264aa11a0f4befef356c03e83f2b1f6eba
Content-length: 4
b
c
Node-path: trunk/b
Node-action: delete
Revision-number: 7
Prop-content-length: 116
Content-length: 116
K 7
svn:log
V 14
change b again
K 10
svn:author
V 7
pmezard
K 8
svn:date
V 27
2010-05-19T20:16:17.070868Z
PROPS-END
Node-path: branches/old/b
Node-kind: file
Node-action: change
Text-content-length: 6
Text-content-md5: cdcfb41554e2d092c13f5e6839e63577
Text-content-sha1: 17ac58cabedebea235d1b5605531d5b1559797e9
Content-length: 6
b
b
b
Revision-number: 8
Prop-content-length: 114
Content-length: 114
K 7
svn:log
V 12
move to old2
K 10
svn:author
V 7
pmezard
K 8
svn:date
V 27
2010-05-19T20:16:20.063098Z
PROPS-END
Node-path: branches/old2
Node-kind: dir
Node-action: add
Node-copyfrom-rev: 7
Node-copyfrom-path: branches/old
Node-path: branches/old
Node-action: delete
Revision-number: 9
Prop-content-length: 118
Content-length: 118
K 7
svn:log
V 16
move back to old
K 10
svn:author
V 7
pmezard
K 8
svn:date
V 27
2010-05-19T20:16:22.062931Z
PROPS-END
Node-path: branches/old
Node-kind: dir
Node-action: add
Node-copyfrom-rev: 8
Node-copyfrom-path: branches/old2
Node-path: branches/old2
Node-action: delete
Revision-number: 10
Prop-content-length: 118
Content-length: 118
K 7
svn:log
V 16
last change to a
K 10
svn:author
V 7
pmezard
K 8
svn:date
V 27
2010-05-19T20:16:23.075562Z
PROPS-END
Node-path: trunk/a
Node-kind: file
Node-action: change
Text-content-length: 2
Text-content-md5: 60b725f10c9c85c70d97880dfe8191b3
Text-content-sha1: 3f786850e387550fdab836ed7e6dc881de23001b
Content-length: 2
a
Revision-number: 11
Prop-content-length: 126
Content-length: 126
K 7
svn:log
V 24
branch trunk@1 into old3
K 10
svn:author
V 7
pmezard
K 8
svn:date
V 27
2010-05-19T20:16:25.107655Z
PROPS-END
Node-path: branches/old3
Node-kind: dir
Node-action: add
Node-copyfrom-rev: 1
Node-copyfrom-path: trunk