annotate tests/svn-safe-append.py @ 50400:95acba2c29f6

encoding: avoid quadratic time complexity when json-encoding non-UTF8 strings Apparently the code uses "+=" with a bytes object, which is linear-time, so the whole encoding is quadratic-time. This patch makes us use a bytearray object, instead, which has a(n amortized-)constant-time append operation. The encoding is still not particularly fast, but at least a 10MB file takes tens of seconds, not many hours to encode.
author Arseniy Alekseyev <aalekseyev@janestreet.com>
date Mon, 06 Mar 2023 11:27:57 +0000
parents 6000f5b25c9b
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
45830
c102b704edb5 global: use python3 in shebangs
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39724
diff changeset
1 #!/usr/bin/env python3
6439
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
2
29195
bdba6a2015d0 py3: make tests/svn-safe-append.py use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents: 6439
diff changeset
3
6439
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
4 __doc__ = """Same as `echo a >> b`, but ensures a changed mtime of b.
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
5 Without this svn will not detect workspace changes."""
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
6
29195
bdba6a2015d0 py3: make tests/svn-safe-append.py use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents: 6439
diff changeset
7 import os
36781
ffa3026d4196 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime
Augie Fackler <augie@google.com>
parents: 29195
diff changeset
8 import stat
29195
bdba6a2015d0 py3: make tests/svn-safe-append.py use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents: 6439
diff changeset
9 import sys
6439
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
10
39724
e1e10cbb5568 py3: make tests/svn-safe-append.py compatible with python 3
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 36781
diff changeset
11 if sys.version_info[0] >= 3:
e1e10cbb5568 py3: make tests/svn-safe-append.py compatible with python 3
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 36781
diff changeset
12 text = os.fsencode(sys.argv[1])
e1e10cbb5568 py3: make tests/svn-safe-append.py compatible with python 3
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 36781
diff changeset
13 fname = os.fsencode(sys.argv[2])
e1e10cbb5568 py3: make tests/svn-safe-append.py compatible with python 3
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 36781
diff changeset
14 else:
e1e10cbb5568 py3: make tests/svn-safe-append.py compatible with python 3
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 36781
diff changeset
15 text = sys.argv[1]
e1e10cbb5568 py3: make tests/svn-safe-append.py compatible with python 3
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 36781
diff changeset
16 fname = sys.argv[2]
6439
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
17
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
18 f = open(fname, "ab")
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
19 try:
36781
ffa3026d4196 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime
Augie Fackler <augie@google.com>
parents: 29195
diff changeset
20 before = os.fstat(f.fileno())[stat.ST_MTIME]
6439
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
21 f.write(text)
39724
e1e10cbb5568 py3: make tests/svn-safe-append.py compatible with python 3
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 36781
diff changeset
22 f.write(b"\n")
6439
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
23 finally:
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
24 f.close()
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
25 inc = 1
36781
ffa3026d4196 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime
Augie Fackler <augie@google.com>
parents: 29195
diff changeset
26 now = os.stat(fname)[stat.ST_MTIME]
6439
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
27 while now == before:
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
28 t = now + inc
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
29 inc += 1
c1b47c0fd2b6 convert: fix test-convert-svn-* problems with mtime not changing
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents:
diff changeset
30 os.utime(fname, (t, t))
36781
ffa3026d4196 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime
Augie Fackler <augie@google.com>
parents: 29195
diff changeset
31 now = os.stat(fname)[stat.ST_MTIME]