view tests/test-check-pytype.t @ 47343:9f798c1b0d89 stable

cext: fix memory leak in phases computation

Without this a buffer whose size in bytes is the number of
changesets in the repository is leaked each time the repository is
opened and changeset phases are computed.

Impact: the current code in hgwebdir creates a new `localrepository`
instance for each HTTP request. Since any pull or push is made of several
requests, a team of 100 people can easily produce thousands of such
requests per day.

Being a low-level malloc, this leak can't be seen with the gc module and
tools relying on that, but was spotted by valgrind immediately.

Reproduction
------------

  for i in range(cl_args.iterations):
      repo = hg.repository(baseui, repo_path)
      rev = repo.revs(rev).first()
      ctx = repo[rev]

  del ctx
  del repo
  # avoid any pollution by other type of leak
  # (that should be fixed in 5.8)
  repoview._filteredrepotypes.clear()

  gc.collect()

Measurements
------------

Resident Set Size (RSS), taken on a clone of
mozilla-central for performance analysis (440 000
changesets).

before:
  5.8+hg19.5ac0f2a8ba72  1000 iterations: 1606MB
  5.8+hg19.5ac0f2a8ba72 10000 iterations: 5723MB
after:
  5.8+hg20.e2084d39e145  1000 iterations:  555MB
  5.8+hg20.e2084d39e145 10000 iterations:  555MB
         (double checked, not a copy/paste error)

(e2084d39e14 is the present changeset, before amendment
of the message to add the measurements)

#require pytype py3 slow

  $ cd $RUNTESTDIR/..

Many of the individual files that are excluded here confuse pytype
because they do a mix of Python 2 and Python 3 things
conditionally. There's no good way to help it out with that as far as
I can tell, so let's just hide those files from it for now. We should
endeavor to empty this list out over time, as some of these are
probably hiding real problems.

mercurial/bundlerepo.py       # no vfs and ui attrs on bundlerepo
mercurial/changegroup.py      # mysterious incorrect type detection
mercurial/chgserver.py        # [attribute-error]
mercurial/cmdutil.py          # No attribute 'markcopied' on mercurial.context.filectx [attribute-error]
mercurial/context.py          # many [attribute-error]
mercurial/copies.py           # No attribute 'items' on None [attribute-error]
mercurial/crecord.py          # tons of [attribute-error], [module-attr]
mercurial/debugcommands.py    # [wrong-arg-types]
mercurial/dispatch.py         # initstdio: No attribute ... on TextIO [attribute-error]
mercurial/exchange.py         # [attribute-error]
mercurial/hgweb/hgweb_mod.py  # [attribute-error], [name-error], [wrong-arg-types]
mercurial/hgweb/server.py     # [attribute-error], [name-error], [module-attr]
mercurial/hgweb/webcommands.py  # [missing-parameter]
mercurial/hgweb/wsgicgi.py    # confused values in os.environ
mercurial/httppeer.py         # [attribute-error], [wrong-arg-types]
mercurial/interfaces          # No attribute 'capabilities' on peer [attribute-error]
mercurial/keepalive.py        # [attribute-error]
mercurial/localrepo.py        # [attribute-error]
mercurial/lsprof.py           # unguarded import
mercurial/manifest.py         # [unsupported-operands], [wrong-arg-types]
mercurial/minirst.py          # [unsupported-operands], [attribute-error]
mercurial/patch.py            # [wrong-arg-types]
mercurial/pure/osutil.py      # [invalid-typevar], [not-callable]
mercurial/pure/parsers.py     # [attribute-error]
mercurial/pycompat.py         # bytes vs str issues
mercurial/repoview.py         # [attribute-error]
mercurial/sslutil.py          # [attribute-error]
mercurial/statprof.py         # bytes vs str on TextIO.write() [wrong-arg-types]
mercurial/testing/storage.py  # tons of [attribute-error]
mercurial/ui.py               # [attribute-error], [wrong-arg-types]
mercurial/unionrepo.py        # ui, svfs, unfiltered [attribute-error]
mercurial/upgrade.py          # line 84, in upgraderepo: No attribute 'discard' on Dict[nothing, nothing] [attribute-error]
mercurial/util.py             # [attribute-error], [wrong-arg-count]
mercurial/utils/procutil.py   # [attribute-error], [module-attr], [bad-return-type]
mercurial/utils/stringutil.py # [module-attr], [wrong-arg-count]
mercurial/utils/memorytop.py  # not 3.6 compatible
mercurial/win32.py            # [not-callable]
mercurial/wireprotoframing.py # [unsupported-operands], [attribute-error], [import-error]
mercurial/wireprotoserver.py  # line 253, in _availableapis: No attribute '__iter__' on Callable[[Any, Any], Any] [attribute-error]
mercurial/wireprotov1peer.py  # [attribute-error]
mercurial/wireprotov1server.py  # BUG?: BundleValueError handler accesses subclass's attrs
mercurial/wireprotov2server.py  # [unsupported-operands], [attribute-error]

TODO: use --no-cache on test server?  Caching the files locally helps during
development, but may be a hinderance for CI testing.

  $ pytype -V 3.6 --keep-going --jobs auto mercurial \
  >    -x mercurial/bundlerepo.py \
  >    -x mercurial/changegroup.py \
  >    -x mercurial/chgserver.py \
  >    -x mercurial/cmdutil.py \
  >    -x mercurial/context.py \
  >    -x mercurial/copies.py \
  >    -x mercurial/crecord.py \
  >    -x mercurial/debugcommands.py \
  >    -x mercurial/dispatch.py \
  >    -x mercurial/exchange.py \
  >    -x mercurial/hgweb/hgweb_mod.py \
  >    -x mercurial/hgweb/server.py \
  >    -x mercurial/hgweb/webcommands.py \
  >    -x mercurial/hgweb/wsgicgi.py \
  >    -x mercurial/httppeer.py \
  >    -x mercurial/interfaces \
  >    -x mercurial/keepalive.py \
  >    -x mercurial/localrepo.py \
  >    -x mercurial/lsprof.py \
  >    -x mercurial/manifest.py \
  >    -x mercurial/minirst.py \
  >    -x mercurial/patch.py \
  >    -x mercurial/pure/osutil.py \
  >    -x mercurial/pure/parsers.py \
  >    -x mercurial/pycompat.py \
  >    -x mercurial/repoview.py \
  >    -x mercurial/sslutil.py \
  >    -x mercurial/statprof.py \
  >    -x mercurial/testing/storage.py \
  >    -x mercurial/thirdparty \
  >    -x mercurial/ui.py \
  >    -x mercurial/unionrepo.py \
  >    -x mercurial/upgrade.py \
  >    -x mercurial/utils/procutil.py \
  >    -x mercurial/utils/stringutil.py \
  >    -x mercurial/utils/memorytop.py \
  >    -x mercurial/win32.py \
  >    -x mercurial/wireprotoframing.py \
  >    -x mercurial/wireprotoserver.py \
  >    -x mercurial/wireprotov1peer.py \
  >    -x mercurial/wireprotov1server.py \
  >    -x mercurial/wireprotov2server.py \
  >  > $TESTTMP/pytype-output.txt || cat $TESTTMP/pytype-output.txt

Only show the results on a failure, because the output on success is also
voluminous and variable.