view tests/test-diff-upgrade.t @ 47010:76ae43d5b1db stable

repoview: fix memory leak of filtered repo classes

The leak occurs in long-running server processes with
extensions, and is measured at 110kB per request.

Before this change, the contents of the `_filteredrepotypes`
cache are not properly garbage collected, despite it begin
a `WeakKeyDictionary`.

Extensions have a tendency to generate a new repository class
for each `localrepo` instantiation. Server processes based
on `hgwebdir_mod` will instantiate a new `localrepo` for each
HTTP request that involves a repository.

As a result, with a testing process that repeatedly opens a
repository with several extensions activated
(`topic` notably among them), we see a steady increase in
resident memory of 110kB per repository instantiation before this
change. This is also true, if we call `gc.collect()` at each
instantiation, like `hgwebdir_mod` does, or not.

The cause of the leak is that the *values* aren't weak references.
This change uses `weakref.ref` for the values, and this makes
in our measurements the resident size increase drop to 5kB per
repository instantiation, with no explicit call of `gc.collect()`
at all.
There is currently no reason to believe that this remaining leak
of 5kB is related to or even due to Mercurial core.

We've also seen evidence that `ui.ui` instances weren't properly
garbage collected before the change (with the change, they are).
This could explain why the figures are relatively high.

In theory, the collection of weak references could lead to
much more misses in the cache, so we measured the impact on
the original case that was motivation for introducing that cache
in 7e89bd0cfb86 (see also issue5043): `hg convert` of the
mozilla-central repository. The bad news here is that there is a
major memory leak there, both with and without the present changeset.
There were no more cache misses, and we could see no
more memory leak with this change: the resident size after importing
roughly 100000 changesets was at 12.4GB before, and 12.5GB after.
The small increase is mentioned for completeness only, and we
believe that it should be ignored, at least as long as the main
leak isn't fixed. At less than 1% of the main leak, even finding out
whether it is merely noise would be wasteful.

Original context where this was spotted and first mitigated:
  https://foss.heptapod.net/heptapod/heptapod/-/issues/466

The leak reduction was also obtained in Heptapod inner HTTP server,
which amounts to the same as `hgwebdir_mod` for these questions.

The measurements done with Python 3.9, similar figures seen with 3.8.
More work on our side would be needed to give measurements with 2.7,
because of testing server process does not support it.

#require execbit

  $ cat <<EOF >> $HGRCPATH
  > [extensions]
  > autodiff = $TESTDIR/autodiff.py
  > [diff]
  > nodates = 1
  > EOF

  $ hg init repo
  $ cd repo



make a combination of new, changed and deleted file

  $ echo regular > regular
  $ echo rmregular > rmregular
  $ "$PYTHON" -c "open('bintoregular', 'wb').write(b'\0')"
  $ touch rmempty
  $ echo exec > exec
  $ chmod +x exec
  $ echo rmexec > rmexec
  $ chmod +x rmexec
  $ echo setexec > setexec
  $ echo unsetexec > unsetexec
  $ chmod +x unsetexec
  $ echo binary > binary
  $ "$PYTHON" -c "open('rmbinary', 'wb').write(b'\0')"
  $ hg ci -Am addfiles
  adding binary
  adding bintoregular
  adding exec
  adding regular
  adding rmbinary
  adding rmempty
  adding rmexec
  adding rmregular
  adding setexec
  adding unsetexec
  $ echo regular >> regular
  $ echo newregular >> newregular
  $ rm rmempty
  $ touch newempty
  $ rm rmregular
  $ echo exec >> exec
  $ echo newexec > newexec
  $ echo bintoregular > bintoregular
  $ chmod +x newexec
  $ rm rmexec
  $ chmod +x setexec
  $ chmod -x unsetexec
  $ "$PYTHON" -c "open('binary', 'wb').write(b'\0\0')"
  $ "$PYTHON" -c "open('newbinary', 'wb').write(b'\0')"
  $ rm rmbinary
  $ hg addremove -s 0
  adding newbinary
  adding newempty
  adding newexec
  adding newregular
  removing rmbinary
  removing rmempty
  removing rmexec
  removing rmregular

git=no: regular diff for all files

  $ hg autodiff --git=no
  diff -r a66d19b9302d binary
  Binary file binary has changed
  diff -r a66d19b9302d bintoregular
  Binary file bintoregular has changed
  diff -r a66d19b9302d exec
  --- a/exec
  +++ b/exec
  @@ -1,1 +1,2 @@
   exec
  +exec
  diff -r a66d19b9302d newbinary
  Binary file newbinary has changed
  diff -r a66d19b9302d newexec
  --- /dev/null
  +++ b/newexec
  @@ -0,0 +1,1 @@
  +newexec
  diff -r a66d19b9302d newregular
  --- /dev/null
  +++ b/newregular
  @@ -0,0 +1,1 @@
  +newregular
  diff -r a66d19b9302d regular
  --- a/regular
  +++ b/regular
  @@ -1,1 +1,2 @@
   regular
  +regular
  diff -r a66d19b9302d rmbinary
  Binary file rmbinary has changed
  diff -r a66d19b9302d rmexec
  --- a/rmexec
  +++ /dev/null
  @@ -1,1 +0,0 @@
  -rmexec
  diff -r a66d19b9302d rmregular
  --- a/rmregular
  +++ /dev/null
  @@ -1,1 +0,0 @@
  -rmregular

git=yes: git diff for single regular file

  $ hg autodiff --git=yes regular
  diff --git a/regular b/regular
  --- a/regular
  +++ b/regular
  @@ -1,1 +1,2 @@
   regular
  +regular

git=auto: regular diff for regular files and non-binary removals

  $ hg autodiff --git=auto regular newregular rmregular rmexec
  diff -r a66d19b9302d newregular
  --- /dev/null
  +++ b/newregular
  @@ -0,0 +1,1 @@
  +newregular
  diff -r a66d19b9302d regular
  --- a/regular
  +++ b/regular
  @@ -1,1 +1,2 @@
   regular
  +regular
  diff -r a66d19b9302d rmexec
  --- a/rmexec
  +++ /dev/null
  @@ -1,1 +0,0 @@
  -rmexec
  diff -r a66d19b9302d rmregular
  --- a/rmregular
  +++ /dev/null
  @@ -1,1 +0,0 @@
  -rmregular

  $ for f in exec newexec setexec unsetexec binary newbinary newempty rmempty rmbinary bintoregular; do
  >     echo
  >     echo '% git=auto: git diff for' $f
  >     hg autodiff --git=auto $f
  > done
  
  % git=auto: git diff for exec
  diff -r a66d19b9302d exec
  --- a/exec
  +++ b/exec
  @@ -1,1 +1,2 @@
   exec
  +exec
  
  % git=auto: git diff for newexec
  diff --git a/newexec b/newexec
  new file mode 100755
  --- /dev/null
  +++ b/newexec
  @@ -0,0 +1,1 @@
  +newexec
  
  % git=auto: git diff for setexec
  diff --git a/setexec b/setexec
  old mode 100644
  new mode 100755
  
  % git=auto: git diff for unsetexec
  diff --git a/unsetexec b/unsetexec
  old mode 100755
  new mode 100644
  
  % git=auto: git diff for binary
  diff --git a/binary b/binary
  index a9128c283485202893f5af379dd9beccb6e79486..09f370e38f498a462e1ca0faa724559b6630c04f
  GIT binary patch
  literal 2
  Jc${Nk0000200961
  
  
  % git=auto: git diff for newbinary
  diff --git a/newbinary b/newbinary
  new file mode 100644
  index 0000000000000000000000000000000000000000..f76dd238ade08917e6712764a16a22005a50573d
  GIT binary patch
  literal 1
  Ic${MZ000310RR91
  
  
  % git=auto: git diff for newempty
  diff --git a/newempty b/newempty
  new file mode 100644
  
  % git=auto: git diff for rmempty
  diff --git a/rmempty b/rmempty
  deleted file mode 100644
  
  % git=auto: git diff for rmbinary
  diff --git a/rmbinary b/rmbinary
  deleted file mode 100644
  index f76dd238ade08917e6712764a16a22005a50573d..0000000000000000000000000000000000000000
  GIT binary patch
  literal 0
  Hc$@<O00001
  
  
  % git=auto: git diff for bintoregular
  diff --git a/bintoregular b/bintoregular
  index f76dd238ade08917e6712764a16a22005a50573d..9c42f2b6427d8bf034b7bc23986152dc01bfd3ab
  GIT binary patch
  literal 13
  Uc$`bh%qz(+N=+}#Ni5<5043uE82|tP
  


git=warn: regular diff with data loss warnings

  $ hg autodiff --git=warn
  diff -r a66d19b9302d binary
  Binary file binary has changed
  diff -r a66d19b9302d bintoregular
  Binary file bintoregular has changed
  diff -r a66d19b9302d exec
  --- a/exec
  +++ b/exec
  @@ -1,1 +1,2 @@
   exec
  +exec
  diff -r a66d19b9302d newbinary
  Binary file newbinary has changed
  diff -r a66d19b9302d newexec
  --- /dev/null
  +++ b/newexec
  @@ -0,0 +1,1 @@
  +newexec
  diff -r a66d19b9302d newregular
  --- /dev/null
  +++ b/newregular
  @@ -0,0 +1,1 @@
  +newregular
  diff -r a66d19b9302d regular
  --- a/regular
  +++ b/regular
  @@ -1,1 +1,2 @@
   regular
  +regular
  diff -r a66d19b9302d rmbinary
  Binary file rmbinary has changed
  diff -r a66d19b9302d rmexec
  --- a/rmexec
  +++ /dev/null
  @@ -1,1 +0,0 @@
  -rmexec
  diff -r a66d19b9302d rmregular
  --- a/rmregular
  +++ /dev/null
  @@ -1,1 +0,0 @@
  -rmregular
  data lost for: binary
  data lost for: bintoregular
  data lost for: newbinary
  data lost for: newempty
  data lost for: newexec
  data lost for: rmbinary
  data lost for: rmempty
  data lost for: setexec
  data lost for: unsetexec

git=abort: fail on execute bit change

  $ hg autodiff --git=abort regular setexec
  abort: losing data for setexec
  [255]

git=abort: succeed on regular file

  $ hg autodiff --git=abort regular
  diff -r a66d19b9302d regular
  --- a/regular
  +++ b/regular
  @@ -1,1 +1,2 @@
   regular
  +regular

  $ cd ..