Mercurial > hg
annotate contrib/python3-ratchet.py @ 40056:324b4b10351e
revlog: rewrite censoring logic
I was able to corrupt a revlog relatively easily with the existing
censoring code. The underlying problem is that the existing code
doesn't fully take delta chains into account. When copying revisions
that occur after the censored revision, the delta base can refer
to a censored revision. Then at read time, things blow up due to the
revision data not being a compressed delta.
This commit rewrites the revlog censoring code to take a higher-level
approach. We now create a new revlog instance pointing at temp files.
We iterate through each revision in the source revlog and insert
those revisions into the new revlog, replacing the censored revision's
data along the way.
The new implementation isn't as efficient as the old one. This is
because it will fully engage delta computation on insertion. But I
don't think it matters.
The new implementation is a bit hacky because it attempts to reload
the revlog instance with a new revlog index/data file. This is fragile.
But this is needed because the index (which could be backed by C) would
have a cached copy of the old, possibly changed data and that could
lead to problems accessing index or revision data later.
One benefit of the new approach is that we integrate with the
transaction. The old revlog is backed up and if the transaction is
rolled back, the original revlog is restored.
As part of this, we had to teach the transaction about the store
vfs. I'm not super keen about this. But this was the easiest way
to hook things up to the transaction. We /could/ just ignore the
transaction like we were doing before. But any file mutation should
be governed by transaction semantics, including undo during rollback.
Differential Revision: https://phab.mercurial-scm.org/D4869
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Tue, 02 Oct 2018 17:34:34 -0700 |
parents | 3790610c2793 |
children | 2372284d9457 |
rev | line source |
---|---|
32855
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
1 # Copyright 2012 Facebook |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
2 # |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
3 # This software may be used and distributed according to the terms of the |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
4 # GNU General Public License version 2 or any later version. |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
5 """Find tests that newly pass under Python 3. |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
6 |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
7 The approach is simple: we maintain a whitelist of Python 3 passing |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
8 tests in the repository, and periodically run all the /other/ tests |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
9 and look for new passes. Any newly passing tests get automatically |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
10 added to the whitelist. |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
11 |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
12 You probably want to run it like this: |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
13 |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
14 $ cd tests |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
15 $ python3 ../contrib/python3-ratchet.py \ |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
16 > --working-tests=../contrib/python3-whitelist |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
17 """ |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
18 from __future__ import print_function |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
19 from __future__ import absolute_import |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
20 |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
21 import argparse |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
22 import json |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
23 import os |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
24 import subprocess |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
25 import sys |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
26 |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
27 _hgenv = dict(os.environ) |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
28 _hgenv.update({ |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
29 'HGPLAIN': '1', |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
30 }) |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
31 |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
32 _HG_FIRST_CHANGE = '9117c6561b0bd7792fa13b50d28239d51b78e51f' |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
33 |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
34 def _runhg(*args): |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
35 return subprocess.check_output(args, env=_hgenv) |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
36 |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
37 def _is_hg_repo(path): |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
38 return _runhg('hg', 'log', '-R', path, |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
39 '-r0', '--template={node}').strip() == _HG_FIRST_CHANGE |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
40 |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
41 def _py3default(): |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
42 if sys.version_info[0] >= 3: |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
43 return sys.executable |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
44 return 'python3' |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
45 |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
46 def main(argv=()): |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
47 p = argparse.ArgumentParser() |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
48 p.add_argument('--working-tests', |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
49 help='List of tests that already work in Python 3.') |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
50 p.add_argument('--commit-to-repo', |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
51 help='If set, commit newly fixed tests to the given repo') |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
52 p.add_argument('-j', default=os.sysconf(r'SC_NPROCESSORS_ONLN'), type=int, |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
53 help='Number of parallel tests to run.') |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
54 p.add_argument('--python3', default=_py3default(), |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
55 help='python3 interpreter to use for test run') |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
56 p.add_argument('--commit-user', |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
57 default='python3-ratchet@mercurial-scm.org', |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
58 help='Username to specify when committing to a repo.') |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
59 opts = p.parse_args(argv) |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
60 if opts.commit_to_repo: |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
61 if not _is_hg_repo(opts.commit_to_repo): |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
62 print('abort: specified repository is not the hg repository') |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
63 sys.exit(1) |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
64 if not opts.working_tests or not os.path.isfile(opts.working_tests): |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
65 print('abort: --working-tests must exist and be a file (got %r)' % |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
66 opts.working_tests) |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
67 sys.exit(1) |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
68 elif opts.commit_to_repo: |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
69 root = _runhg('hg', 'root').strip() |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
70 if not opts.working_tests.startswith(root): |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
71 print('abort: if --commit-to-repo is given, ' |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
72 '--working-tests must be from that repo') |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
73 sys.exit(1) |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
74 try: |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
75 subprocess.check_call([opts.python3, '-c', |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
76 'import sys ; ' |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
77 'assert ((3, 5) <= sys.version_info < (3, 6) ' |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
78 'or sys.version_info >= (3, 6, 2))']) |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
79 except subprocess.CalledProcessError: |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
80 print('warning: Python 3.6.0 and 3.6.1 have ' |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
81 'a bug which breaks Mercurial') |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
82 print('(see https://bugs.python.org/issue29714 for details)') |
36589
3790610c2793
py3: bail on ratcheting tests forward on 3.6.0 and 3.6.1
Kevin Bullock <kbullock+mercurial@ringworld.org>
parents:
32855
diff
changeset
|
83 sys.exit(1) |
32855
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
84 |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
85 rt = subprocess.Popen([opts.python3, 'run-tests.py', '-j', str(opts.j), |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
86 '--blacklist', opts.working_tests, '--json']) |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
87 rt.wait() |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
88 with open('report.json') as f: |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
89 data = f.read() |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
90 report = json.loads(data.split('=', 1)[1]) |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
91 newpass = set() |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
92 for test, result in report.items(): |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
93 if result['result'] != 'success': |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
94 continue |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
95 # A new passing test! Huzzah! |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
96 newpass.add(test) |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
97 if newpass: |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
98 # We already validated the repo, so we can just dive right in |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
99 # and commit. |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
100 if opts.commit_to_repo: |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
101 print(len(newpass), 'new passing tests on Python 3!') |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
102 with open(opts.working_tests) as f: |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
103 oldpass = {l for l in f.read().splitlines() if l} |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
104 with open(opts.working_tests, 'w') as f: |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
105 for p in sorted(oldpass | newpass): |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
106 f.write('%s\n' % p) |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
107 _runhg('hg', 'commit', '-R', opts.commit_to_repo, |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
108 '--user', opts.commit_user, |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
109 '--message', 'python3: expand list of passing tests') |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
110 else: |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
111 print('Newly passing tests:', '\n'.join(sorted(newpass))) |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
112 sys.exit(2) |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
113 |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
114 if __name__ == '__main__': |
28f429d19a71
contrib: add a ratchet for tests in Python 3
Augie Fackler <augie@google.com>
parents:
diff
changeset
|
115 main(sys.argv[1:]) |