Mercurial > hg
view tests/filterpyflakes.py @ 50400:95acba2c29f6
encoding: avoid quadratic time complexity when json-encoding non-UTF8 strings
Apparently the code uses "+=" with a bytes object, which is linear-time, so the
whole encoding is quadratic-time. This patch makes us use a bytearray object,
instead, which has a(n amortized-)constant-time append operation.
The encoding is still not particularly fast, but at least a 10MB file
takes tens of seconds, not many hours to encode.
author | Arseniy Alekseyev <aalekseyev@janestreet.com> |
---|---|
date | Mon, 06 Mar 2023 11:27:57 +0000 |
parents | 6000f5b25c9b |
children | e07dc1e7a454 |
line wrap: on
line source
#!/usr/bin/env python3 # Filter output by pyflakes to control which warnings we check import re import sys lines = [] for line in sys.stdin: # We blacklist tests that are too noisy for us pats = [ r"undefined name 'WindowsError'", r"redefinition of unused '[^']+' from line", # for cffi, allow re-exports from pure.* r"cffi/[^:]*:.*\bimport \*' used", r"cffi/[^:]*:.*\*' imported but unused", ] keep = True for pat in pats: if re.search(pat, line): keep = False break # pattern matches if keep: fn = line.split(':', 1)[0] f = open(fn) data = f.read() f.close() if 'no-' 'check-code' in data: continue lines.append(line) for line in lines: sys.stdout.write(line) print()