annotate contrib/import-checker.py @ 35757:bbf7abd09ff0

streamclone: rework canperformstreamclone There is code about bundle2 laying around in `canperformstreamclone` but not put to any uses. As we discovered with the previous patch, streambundle 'v1' won't work on bundle2 because they are readline based. So we jump to 'v2' as the first expected supported version.
author Boris Feld <boris.feld@octobus.net>
date Thu, 18 Jan 2018 00:45:27 +0100
parents 41401f502c83
children 922b3fae9c7d
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
26954
f804bf27439b import-checker: make it executable for convenience
Yuya Nishihara <yuya@tcha.org>
parents: 26781
diff changeset
1 #!/usr/bin/env python
f804bf27439b import-checker: make it executable for convenience
Yuya Nishihara <yuya@tcha.org>
parents: 26781
diff changeset
2
28703
a274c4f9087a py3: use print_function in import-checker
timeless <timeless@mozdev.org>
parents: 28702
diff changeset
3 from __future__ import absolute_import, print_function
28702
e44f671018e3 py3: use absolute_import in import-checker
timeless <timeless@mozdev.org>
parents: 28700
diff changeset
4
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
5 import ast
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
6 import collections
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
7 import os
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
8 import re
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
9 import sys
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
10
20198
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
11 # Import a minimal set of stdlib modules needed for list_stdlib_modules()
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
12 # to work when run from a virtualenv. The modules were chosen empirically
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
13 # so that the return value matches the return value without virtualenv.
29211
b42c2a66a698 py3: make contrib/import-checker.py get along with itself
Yuya Nishihara <yuya@tcha.org>
parents: 29208
diff changeset
14 if True: # disable lexical sorting checks
33895
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32621
diff changeset
15 try:
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32621
diff changeset
16 import BaseHTTPServer as basehttpserver
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32621
diff changeset
17 except ImportError:
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32621
diff changeset
18 basehttpserver = None
29211
b42c2a66a698 py3: make contrib/import-checker.py get along with itself
Yuya Nishihara <yuya@tcha.org>
parents: 29208
diff changeset
19 import zlib
20198
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
20
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
21 # Whitelist of modules that symbols can be directly imported from.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
22 allowsymbolimports = (
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
23 '__future__',
33937
5ed0be4d9df9 contrib: add bzrlib to list of packages from which we import symbols
Augie Fackler <raf@durin42.com>
parents: 33933
diff changeset
24 'bzrlib',
33933
2d64b2f1787b contrib: allow symbol imports from hgclient for tests
Augie Fackler <raf@durin42.com>
parents: 33915
diff changeset
25 'hgclient',
33911
c9cf69d0c3b9 contrib: allow importing "symbols" from mercurial
Augie Fackler <raf@durin42.com>
parents: 33908
diff changeset
26 'mercurial',
27018
e5be48dd8215 import-checker: allow symbol imports from hgweb.common and .request
Yuya Nishihara <yuya@tcha.org>
parents: 26965
diff changeset
27 'mercurial.hgweb.common',
e5be48dd8215 import-checker: allow symbol imports from hgweb.common and .request
Yuya Nishihara <yuya@tcha.org>
parents: 26965
diff changeset
28 'mercurial.hgweb.request',
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
29 'mercurial.i18n',
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
30 'mercurial.node',
32540
95085d747db8 import-checker: allow importing symbols from pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32458
diff changeset
31 # for cffi modules to re-export pure functions
95085d747db8 import-checker: allow importing symbols from pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32458
diff changeset
32 'mercurial.pure.base85',
95085d747db8 import-checker: allow importing symbols from pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32458
diff changeset
33 'mercurial.pure.bdiff',
95085d747db8 import-checker: allow importing symbols from pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32458
diff changeset
34 'mercurial.pure.diffhelpers',
95085d747db8 import-checker: allow importing symbols from pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32458
diff changeset
35 'mercurial.pure.mpatch',
95085d747db8 import-checker: allow importing symbols from pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32458
diff changeset
36 'mercurial.pure.osutil',
95085d747db8 import-checker: allow importing symbols from pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32458
diff changeset
37 'mercurial.pure.parsers',
34396
41401f502c83 tests: disable lints on mercurial/thirdparty
Siddharth Agarwal <sid0@fb.com>
parents: 34055
diff changeset
38 # third-party imports should be directly imported
41401f502c83 tests: disable lints on mercurial/thirdparty
Siddharth Agarwal <sid0@fb.com>
parents: 34055
diff changeset
39 'mercurial.thirdparty',
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
40 )
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
41
32457
d02888308235 import-checker: add a way to directly import certain symbols
Siddharth Agarwal <sid0@fb.com>
parents: 32413
diff changeset
42 # Whitelist of symbols that can be directly imported.
32458
0906b85bf222 demandimport: move to separate package
Siddharth Agarwal <sid0@fb.com>
parents: 32457
diff changeset
43 directsymbols = (
0906b85bf222 demandimport: move to separate package
Siddharth Agarwal <sid0@fb.com>
parents: 32457
diff changeset
44 'demandimport',
0906b85bf222 demandimport: move to separate package
Siddharth Agarwal <sid0@fb.com>
parents: 32457
diff changeset
45 )
32457
d02888308235 import-checker: add a way to directly import certain symbols
Siddharth Agarwal <sid0@fb.com>
parents: 32413
diff changeset
46
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
47 # Modules that must be aliased because they are commonly confused with
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
48 # common variables and can create aliasing and readability issues.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
49 requirealias = {
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
50 'ui': 'uimod',
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
51 }
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
52
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
53 def usingabsolute(root):
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
54 """Whether absolute imports are being used."""
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
55 if sys.version_info[0] >= 3:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
56 return True
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
57
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
58 for node in ast.walk(root):
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
59 if isinstance(node, ast.ImportFrom):
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
60 if node.module == '__future__':
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
61 for n in node.names:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
62 if n.name == 'absolute_import':
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
63 return True
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
64
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
65 return False
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
66
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
67 def walklocal(root):
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
68 """Recursively yield all descendant nodes but not in a different scope"""
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
69 todo = collections.deque(ast.iter_child_nodes(root))
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
70 yield root, False
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
71 while todo:
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
72 node = todo.popleft()
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
73 newscope = isinstance(node, ast.FunctionDef)
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
74 if not newscope:
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
75 todo.extend(ast.iter_child_nodes(node))
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
76 yield node, newscope
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
77
32413
194b0f781132 import-checker: drop workaround for pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32411
diff changeset
78 def dotted_name_of_path(path):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
79 """Given a relative path to a source file, return its dotted module name.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
80
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
81 >>> dotted_name_of_path('mercurial/error.py')
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
82 'mercurial.error'
20383
4990abb4729d import-checker: fix names of dynamically loaded modules
Mads Kiilerich <madski@unity3d.com>
parents: 20238
diff changeset
83 >>> dotted_name_of_path('zlibmodule.so')
4990abb4729d import-checker: fix names of dynamically loaded modules
Mads Kiilerich <madski@unity3d.com>
parents: 20238
diff changeset
84 'zlib'
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
85 """
27620
0c60843b55b5 import-checker: normalize directory separator to get module name on Windows
Yuya Nishihara <yuya@tcha.org>
parents: 27520
diff changeset
86 parts = path.replace(os.sep, '/').split('/')
20391
466e4c574db0 import-checker: handle standard modules with arch in the filename
Mads Kiilerich <madski@unity3d.com>
parents: 20386
diff changeset
87 parts[-1] = parts[-1].split('.', 1)[0] # remove .py and .so and .ARCH.so
20383
4990abb4729d import-checker: fix names of dynamically loaded modules
Mads Kiilerich <madski@unity3d.com>
parents: 20238
diff changeset
88 if parts[-1].endswith('module'):
4990abb4729d import-checker: fix names of dynamically loaded modules
Mads Kiilerich <madski@unity3d.com>
parents: 20238
diff changeset
89 parts[-1] = parts[-1][:-6]
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
90 return '.'.join(parts)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
91
25173
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
92 def fromlocalfunc(modulename, localmods):
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
93 """Get a function to examine which locally defined module the
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
94 target source imports via a specified name.
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
95
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
96 `modulename` is an `dotted_name_of_path()`-ed source file path,
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
97 which may have `.__init__` at the end of it, of the target source.
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
98
32541
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32540
diff changeset
99 `localmods` is a set of absolute `dotted_name_of_path()`-ed source file
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32540
diff changeset
100 paths of locally defined (= Mercurial specific) modules.
25173
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
101
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
102 This function assumes that module names not existing in
26781
1aee2ab0f902 spelling: trivial spell checking
Mads Kiilerich <madski@unity3d.com>
parents: 26221
diff changeset
103 `localmods` are from the Python standard library.
25173
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
104
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
105 This function returns the function, which takes `name` argument,
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
106 and returns `(absname, dottedpath, hassubmod)` tuple if `name`
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
107 matches against locally defined module. Otherwise, it returns
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
108 False.
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
109
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
110 It is assumed that `name` doesn't have `.__init__`.
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
111
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
112 `absname` is an absolute module name of specified `name`
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
113 (e.g. "hgext.convert"). This can be used to compose prefix for sub
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
114 modules or so.
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
115
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
116 `dottedpath` is a `dotted_name_of_path()`-ed source file path
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
117 (e.g. "hgext.convert.__init__") of `name`. This is used to look
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
118 module up in `localmods` again.
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
119
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
120 `hassubmod` is whether it may have sub modules under it (for
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
121 convenient, even though this is also equivalent to "absname !=
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
122 dottednpath")
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
123
32541
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32540
diff changeset
124 >>> localmods = {'foo.__init__', 'foo.foo1',
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32540
diff changeset
125 ... 'foo.bar.__init__', 'foo.bar.bar1',
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32540
diff changeset
126 ... 'baz.__init__', 'baz.baz1'}
25173
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
127 >>> fromlocal = fromlocalfunc('foo.xxx', localmods)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
128 >>> # relative
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
129 >>> fromlocal('foo1')
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
130 ('foo.foo1', 'foo.foo1', False)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
131 >>> fromlocal('bar')
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
132 ('foo.bar', 'foo.bar.__init__', True)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
133 >>> fromlocal('bar.bar1')
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
134 ('foo.bar.bar1', 'foo.bar.bar1', False)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
135 >>> # absolute
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
136 >>> fromlocal('baz')
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
137 ('baz', 'baz.__init__', True)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
138 >>> fromlocal('baz.baz1')
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
139 ('baz.baz1', 'baz.baz1', False)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
140 >>> # unknown = maybe standard library
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
141 >>> fromlocal('os')
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
142 False
25701
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
143 >>> fromlocal(None, 1)
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
144 ('foo', 'foo.__init__', True)
29122
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
145 >>> fromlocal('foo1', 1)
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
146 ('foo.foo1', 'foo.foo1', False)
25701
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
147 >>> fromlocal2 = fromlocalfunc('foo.xxx.yyy', localmods)
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
148 >>> fromlocal2(None, 2)
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
149 ('foo', 'foo.__init__', True)
29122
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
150 >>> fromlocal2('bar2', 1)
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
151 False
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
152 >>> fromlocal2('bar', 2)
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
153 ('foo.bar', 'foo.bar.__init__', True)
25173
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
154 """
33908
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names
Augie Fackler <raf@durin42.com>
parents: 33896
diff changeset
155 if not isinstance(modulename, str):
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names
Augie Fackler <raf@durin42.com>
parents: 33896
diff changeset
156 modulename = modulename.decode('ascii')
25173
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
157 prefix = '.'.join(modulename.split('.')[:-1])
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
158 if prefix:
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
159 prefix += '.'
25701
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
160 def fromlocal(name, level=0):
29374
7712fcde2d56 import-checker: increase portability for python 2.6.x
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29234
diff changeset
161 # name is false value when relative imports are used.
7712fcde2d56 import-checker: increase portability for python 2.6.x
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29234
diff changeset
162 if not name:
25701
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
163 # If relative imports are used, level must not be absolute.
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
164 assert level > 0
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
165 candidates = ['.'.join(modulename.split('.')[:-level])]
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
166 else:
29122
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
167 if not level:
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
168 # Check relative name first.
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
169 candidates = [prefix + name, name]
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
170 else:
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
171 candidates = ['.'.join(modulename.split('.')[:-level]) +
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
172 '.' + name]
25701
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
173
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
174 for n in candidates:
25173
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
175 if n in localmods:
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
176 return (n, n, False)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
177 dottedpath = n + '.__init__'
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
178 if dottedpath in localmods:
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
179 return (n, dottedpath, True)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
180 return False
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
181 return fromlocal
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
182
32542
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32541
diff changeset
183 def populateextmods(localmods):
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32541
diff changeset
184 """Populate C extension modules based on pure modules"""
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32541
diff changeset
185 newlocalmods = set(localmods)
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32541
diff changeset
186 for n in localmods:
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32541
diff changeset
187 if n.startswith('mercurial.pure.'):
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32541
diff changeset
188 m = n[len('mercurial.pure.'):]
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32541
diff changeset
189 newlocalmods.add('mercurial.cext.' + m)
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32541
diff changeset
190 newlocalmods.add('mercurial.cffi._' + m)
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32541
diff changeset
191 return newlocalmods
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32541
diff changeset
192
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
193 def list_stdlib_modules():
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
194 """List the modules present in the stdlib.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
195
33895
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32621
diff changeset
196 >>> py3 = sys.version_info[0] >= 3
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
197 >>> mods = set(list_stdlib_modules())
33895
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32621
diff changeset
198 >>> 'BaseHTTPServer' in mods or py3
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
199 True
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
200
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
201 os.path isn't really a module, so it's missing:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
202
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
203 >>> 'os.path' in mods
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
204 False
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
205
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
206 sys requires special treatment, because it's baked into the
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
207 interpreter, but it should still appear:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
208
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
209 >>> 'sys' in mods
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
210 True
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
211
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
212 >>> 'collections' in mods
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
213 True
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
214
33895
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32621
diff changeset
215 >>> 'cStringIO' in mods or py3
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
216 True
29395
4c8026babe8c import-checker: ensure cffi is always a system module
Augie Fackler <raf@durin42.com>
parents: 29374
diff changeset
217
4c8026babe8c import-checker: ensure cffi is always a system module
Augie Fackler <raf@durin42.com>
parents: 29374
diff changeset
218 >>> 'cffi' in mods
4c8026babe8c import-checker: ensure cffi is always a system module
Augie Fackler <raf@durin42.com>
parents: 29374
diff changeset
219 True
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
220 """
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
221 for m in sys.builtin_module_names:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
222 yield m
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
223 # These modules only exist on windows, but we should always
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
224 # consider them stdlib.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
225 for m in ['msvcrt', '_winreg']:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
226 yield m
33912
c856cb1c29be contrib: inform import checker that __builtin__ is a thing
Augie Fackler <raf@durin42.com>
parents: 33911
diff changeset
227 yield '__builtin__'
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
228 yield 'builtins' # python3 only
33915
b4294ff13392 contrib: always treat importlib.* as stdlib
Augie Fackler <raf@durin42.com>
parents: 33912
diff changeset
229 yield 'importlib.abc' # python3 only
b4294ff13392 contrib: always treat importlib.* as stdlib
Augie Fackler <raf@durin42.com>
parents: 33912
diff changeset
230 yield 'importlib.machinery' # python3 only
b4294ff13392 contrib: always treat importlib.* as stdlib
Augie Fackler <raf@durin42.com>
parents: 33912
diff changeset
231 yield 'importlib.util' # python3 only
24669
fbdbff1b486a import-checker: force 'fcntl', 'grp', 'pwd', and 'termios' to stdlib modules
Matt Harbison <matt_harbison@yahoo.com>
parents: 24668
diff changeset
232 for m in 'fcntl', 'grp', 'pwd', 'termios': # Unix only
fbdbff1b486a import-checker: force 'fcntl', 'grp', 'pwd', and 'termios' to stdlib modules
Matt Harbison <matt_harbison@yahoo.com>
parents: 24668
diff changeset
233 yield m
28713
806d260c6f3b tests: fix builtin module test on pypy
Maciej Fijalkowski <fijall@gmail.com>
parents: 28704
diff changeset
234 for m in 'cPickle', 'datetime': # in Python (not C) on PyPy
806d260c6f3b tests: fix builtin module test on pypy
Maciej Fijalkowski <fijall@gmail.com>
parents: 28704
diff changeset
235 yield m
29395
4c8026babe8c import-checker: ensure cffi is always a system module
Augie Fackler <raf@durin42.com>
parents: 29374
diff changeset
236 for m in ['cffi']:
4c8026babe8c import-checker: ensure cffi is always a system module
Augie Fackler <raf@durin42.com>
parents: 29374
diff changeset
237 yield m
32331
bd872f64a8ba cleanup: use set literals
Martin von Zweigbergk <martinvonz@google.com>
parents: 32252
diff changeset
238 stdlib_prefixes = {sys.prefix, sys.exec_prefix}
20198
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
239 # We need to supplement the list of prefixes for the search to work
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
240 # when run from within a virtualenv.
33895
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32621
diff changeset
241 for mod in (basehttpserver, zlib):
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32621
diff changeset
242 if mod is None:
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32621
diff changeset
243 continue
20198
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
244 try:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
245 # Not all module objects have a __file__ attribute.
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
246 filename = mod.__file__
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
247 except AttributeError:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
248 continue
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
249 dirname = os.path.dirname(filename)
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
250 for prefix in stdlib_prefixes:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
251 if dirname.startswith(prefix):
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
252 # Then this directory is redundant.
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
253 break
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
254 else:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
255 stdlib_prefixes.add(dirname)
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
256 for libpath in sys.path:
20201
bc3b48b0f5c8 import-checker: suppress check-code about any()
Augie Fackler <raf@durin42.com>
parents: 20200
diff changeset
257 # We want to walk everything in sys.path that starts with
28700
35ad5bcdeb7e py24: remove check-code py24 notation
timeless <timeless@mozdev.org>
parents: 28400
diff changeset
258 # something in stdlib_prefixes.
35ad5bcdeb7e py24: remove check-code py24 notation
timeless <timeless@mozdev.org>
parents: 28400
diff changeset
259 if not any(libpath.startswith(p) for p in stdlib_prefixes):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
260 continue
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
261 for top, dirs, files in os.walk(libpath):
25733
f99c066f5f9a import-checker: recurse into subtree of sys.path only if __init__.py exists
Yuya Nishihara <yuya@tcha.org>
parents: 25731
diff changeset
262 for i, d in reversed(list(enumerate(dirs))):
25734
9086d0c1def3 import-checker: exclude mercurial packages installed into the system path
Yuya Nishihara <yuya@tcha.org>
parents: 25733
diff changeset
263 if (not os.path.exists(os.path.join(top, d, '__init__.py'))
32621
9e46627baa3c import-checker: add hgdemandimport to local modules
Siddharth Agarwal <sid0@fb.com>
parents: 32542
diff changeset
264 or top == libpath and d in ('hgdemandimport', 'hgext',
9e46627baa3c import-checker: add hgdemandimport to local modules
Siddharth Agarwal <sid0@fb.com>
parents: 32542
diff changeset
265 'mercurial')):
25733
f99c066f5f9a import-checker: recurse into subtree of sys.path only if __init__.py exists
Yuya Nishihara <yuya@tcha.org>
parents: 25731
diff changeset
266 del dirs[i]
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
267 for name in files:
26221
ae65b1b4cb46 import-checker: use modern .endswith for multiple suffixes
Augie Fackler <augie@google.com>
parents: 26166
diff changeset
268 if not name.endswith(('.py', '.so', '.pyc', '.pyo', '.pyd')):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
269 continue
27621
39845b064041 import-checker: list package directory as stdlib module
Yuya Nishihara <yuya@tcha.org>
parents: 27620
diff changeset
270 if name.startswith('__init__.py'):
39845b064041 import-checker: list package directory as stdlib module
Yuya Nishihara <yuya@tcha.org>
parents: 27620
diff changeset
271 full_path = top
39845b064041 import-checker: list package directory as stdlib module
Yuya Nishihara <yuya@tcha.org>
parents: 27620
diff changeset
272 else:
39845b064041 import-checker: list package directory as stdlib module
Yuya Nishihara <yuya@tcha.org>
parents: 27620
diff changeset
273 full_path = os.path.join(top, name)
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
274 rel_path = full_path[len(libpath) + 1:]
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
275 mod = dotted_name_of_path(rel_path)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
276 yield mod
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
277
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
278 stdlib_modules = set(list_stdlib_modules())
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
279
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
280 def imported_modules(source, modulename, f, localmods, ignore_nested=False):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
281 """Given the source of a file as a string, yield the names
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
282 imported by that file.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
283
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
284 Args:
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
285 source: The python source to examine as a string.
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
286 modulename: of specified python source (may have `__init__`)
32541
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32540
diff changeset
287 localmods: set of locally defined module names (may have `__init__`)
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
288 ignore_nested: If true, import statements that do not start in
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
289 column zero will be ignored.
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
290
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
291 Returns:
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
292 A list of absolute module names imported by the given source.
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
293
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
294 >>> f = 'foo/xxx.py'
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
295 >>> modulename = 'foo.xxx'
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
296 >>> localmods = {'foo.__init__': True,
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
297 ... 'foo.foo1': True, 'foo.foo2': True,
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
298 ... 'foo.bar.__init__': True, 'foo.bar.bar1': True,
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
299 ... 'baz.__init__': True, 'baz.baz1': True }
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
300 >>> # standard library (= not locally defined ones)
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
301 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
302 ... 'from stdlib1 import foo, bar; import stdlib2',
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
303 ... modulename, f, localmods))
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
304 []
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
305 >>> # relative importing
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
306 >>> sorted(imported_modules(
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
307 ... 'import foo1; from bar import bar1',
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
308 ... modulename, f, localmods))
26964
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
309 ['foo.bar.bar1', 'foo.foo1']
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
310 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
311 ... 'from bar.bar1 import name1, name2, name3',
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
312 ... modulename, f, localmods))
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
313 ['foo.bar.bar1']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
314 >>> # absolute importing
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
315 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
316 ... 'from baz import baz1, name1',
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
317 ... modulename, f, localmods))
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
318 ['baz.__init__', 'baz.baz1']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
319 >>> # mixed importing, even though it shouldn't be recommended
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
320 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
321 ... 'import stdlib, foo1, baz',
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
322 ... modulename, f, localmods))
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
323 ['baz.__init__', 'foo.foo1']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
324 >>> # ignore_nested
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
325 >>> sorted(imported_modules(
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
326 ... '''import foo
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
327 ... def wat():
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
328 ... import bar
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
329 ... ''', modulename, f, localmods))
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
330 ['foo.__init__', 'foo.bar.__init__']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
331 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
332 ... '''import foo
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
333 ... def wat():
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
334 ... import bar
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
335 ... ''', modulename, f, localmods, ignore_nested=True))
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
336 ['foo.__init__']
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
337 """
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
338 fromlocal = fromlocalfunc(modulename, localmods)
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
339 for node in ast.walk(ast.parse(source, f)):
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
340 if ignore_nested and getattr(node, 'col_offset', 0) > 0:
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
341 continue
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
342 if isinstance(node, ast.Import):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
343 for n in node.names:
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
344 found = fromlocal(n.name)
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
345 if not found:
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
346 # this should import standard library
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
347 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
348 yield found[1]
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
349 elif isinstance(node, ast.ImportFrom):
25701
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
350 found = fromlocal(node.module, node.level)
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
351 if not found:
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
352 # this should import standard library
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
353 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
354
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
355 absname, dottedpath, hassubmod = found
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
356 if not hassubmod:
26964
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
357 # "dottedpath" is not a package; must be imported
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
358 yield dottedpath
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
359 # examination of "node.names" should be redundant
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
360 # e.g.: from mercurial.node import nullid, nullrev
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
361 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
362
26964
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
363 modnotfound = False
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
364 prefix = absname + '.'
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
365 for n in node.names:
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
366 found = fromlocal(prefix + n.name)
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
367 if not found:
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
368 # this should be a function or a property of "node.module"
26964
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
369 modnotfound = True
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
370 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
371 yield found[1]
26964
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
372 if modnotfound:
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
373 # "dottedpath" is a package, but imported because of non-module
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
374 # lookup
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
375 yield dottedpath
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
376
27272
69308357ecd1 import-checker: allow absolute imports of sub modules from local packages
Yuya Nishihara <yuya@tcha.org>
parents: 27018
diff changeset
377 def verify_import_convention(module, source, localmods):
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
378 """Verify imports match our established coding convention.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
379
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
380 We have 2 conventions: legacy and modern. The modern convention is in
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
381 effect when using absolute imports.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
382
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
383 The legacy convention only looks for mixed imports. The modern convention
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
384 is much more thorough.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
385 """
25702
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
386 root = ast.parse(source)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
387 absolute = usingabsolute(root)
25702
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
388
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
389 if absolute:
27272
69308357ecd1 import-checker: allow absolute imports of sub modules from local packages
Yuya Nishihara <yuya@tcha.org>
parents: 27018
diff changeset
390 return verify_modern_convention(module, root, localmods)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
391 else:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
392 return verify_stdlib_on_own_line(root)
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
393
27272
69308357ecd1 import-checker: allow absolute imports of sub modules from local packages
Yuya Nishihara <yuya@tcha.org>
parents: 27018
diff changeset
394 def verify_modern_convention(module, root, localmods, root_col_offset=0):
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
395 """Verify a file conforms to the modern import convention rules.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
396
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
397 The rules of the modern convention are:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
398
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
399 * Ordering is stdlib followed by local imports. Each group is lexically
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
400 sorted.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
401 * Importing multiple modules via "import X, Y" is not allowed: use
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
402 separate import statements.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
403 * Importing multiple modules via "from X import ..." is allowed if using
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
404 parenthesis and one entry per line.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
405 * Only 1 relative import statement per import level ("from .", "from ..")
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
406 is allowed.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
407 * Relative imports from higher levels must occur before lower levels. e.g.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
408 "from .." must be before "from .".
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
409 * Imports from peer packages should use relative import (e.g. do not
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
410 "import mercurial.foo" from a "mercurial.*" module).
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
411 * Symbols can only be imported from specific modules (see
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
412 `allowsymbolimports`). For other modules, first import the module then
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
413 assign the symbol to a module-level variable. In addition, these imports
29208
cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules
Yuya Nishihara <yuya@tcha.org>
parents: 29207
diff changeset
414 must be performed before other local imports. This rule only
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
415 applies to import statements outside of any blocks.
34055
bc2535238de2 import-checker: allow relative import a module being checked
Jun Wu <quark@fb.com>
parents: 33938
diff changeset
416 * Relative imports from the standard library are not allowed, unless that
bc2535238de2 import-checker: allow relative import a module being checked
Jun Wu <quark@fb.com>
parents: 33938
diff changeset
417 library is also a local module.
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
418 * Certain modules must be aliased to alternate names to avoid aliasing
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
419 and readability problems. See `requirealias`.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
420 """
33908
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names
Augie Fackler <raf@durin42.com>
parents: 33896
diff changeset
421 if not isinstance(module, str):
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names
Augie Fackler <raf@durin42.com>
parents: 33896
diff changeset
422 module = module.decode('ascii')
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
423 topmodule = module.split('.')[0]
27272
69308357ecd1 import-checker: allow absolute imports of sub modules from local packages
Yuya Nishihara <yuya@tcha.org>
parents: 27018
diff changeset
424 fromlocal = fromlocalfunc(module, localmods)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
425
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
426 # Whether a local/non-stdlib import has been performed.
28330
f3fb24e36d61 import-checker: report local with stdlib late warning
timeless <timeless@mozdev.org>
parents: 27621
diff changeset
427 seenlocal = None
29208
cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules
Yuya Nishihara <yuya@tcha.org>
parents: 29207
diff changeset
428 # Whether a local/non-stdlib, non-symbol import has been seen.
cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules
Yuya Nishihara <yuya@tcha.org>
parents: 29207
diff changeset
429 seennonsymbollocal = False
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
430 # The last name to be imported (for sorting).
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
431 lastname = None
30595
74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29395
diff changeset
432 laststdlib = None
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
433 # Relative import levels encountered so far.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
434 seenlevels = set()
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
435
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
436 for node, newscope in walklocal(root):
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
437 def msg(fmt, *args):
26956
4b56214ebb7a import-checker: include lineno in warning message
Yuya Nishihara <yuya@tcha.org>
parents: 26955
diff changeset
438 return (fmt % args, node.lineno)
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
439 if newscope:
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
440 # Check for local imports in function
27272
69308357ecd1 import-checker: allow absolute imports of sub modules from local packages
Yuya Nishihara <yuya@tcha.org>
parents: 27018
diff changeset
441 for r in verify_modern_convention(module, node, localmods,
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
442 node.col_offset + 4):
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
443 yield r
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
444 elif isinstance(node, ast.Import):
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
445 # Disallow "import foo, bar" and require separate imports
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
446 # for each module.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
447 if len(node.names) > 1:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
448 yield msg('multiple imported names: %s',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
449 ', '.join(n.name for n in node.names))
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
450
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
451 name = node.names[0].name
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
452 asname = node.names[0].asname
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
453
30595
74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29395
diff changeset
454 stdlib = name in stdlib_modules
74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29395
diff changeset
455
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
456 # Ignore sorting rules on imports inside blocks.
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
457 if node.col_offset == root_col_offset:
30595
74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29395
diff changeset
458 if lastname and name < lastname and laststdlib == stdlib:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
459 yield msg('imports not lexically sorted: %s < %s',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
460 name, lastname)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
461
30595
74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29395
diff changeset
462 lastname = name
74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29395
diff changeset
463 laststdlib = stdlib
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
464
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
465 # stdlib imports should be before local imports.
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
466 if stdlib and seenlocal and node.col_offset == root_col_offset:
28330
f3fb24e36d61 import-checker: report local with stdlib late warning
timeless <timeless@mozdev.org>
parents: 27621
diff changeset
467 yield msg('stdlib import "%s" follows local import: %s',
f3fb24e36d61 import-checker: report local with stdlib late warning
timeless <timeless@mozdev.org>
parents: 27621
diff changeset
468 name, seenlocal)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
469
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
470 if not stdlib:
28330
f3fb24e36d61 import-checker: report local with stdlib late warning
timeless <timeless@mozdev.org>
parents: 27621
diff changeset
471 seenlocal = name
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
472
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
473 # Import of sibling modules should use relative imports.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
474 topname = name.split('.')[0]
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
475 if topname == topmodule:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
476 yield msg('import should be relative: %s', name)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
477
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
478 if name in requirealias and asname != requirealias[name]:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
479 yield msg('%s module must be "as" aliased to %s',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
480 name, requirealias[name])
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
481
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
482 elif isinstance(node, ast.ImportFrom):
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
483 # Resolve the full imported module name.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
484 if node.level > 0:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
485 fullname = '.'.join(module.split('.')[:-node.level])
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
486 if node.module:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
487 fullname += '.%s' % node.module
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
488 else:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
489 assert node.module
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
490 fullname = node.module
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
491
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
492 topname = fullname.split('.')[0]
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
493 if topname == topmodule:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
494 yield msg('import should be relative: %s', fullname)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
495
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
496 # __future__ is special since it needs to come first and use
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
497 # symbol import.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
498 if fullname != '__future__':
34055
bc2535238de2 import-checker: allow relative import a module being checked
Jun Wu <quark@fb.com>
parents: 33938
diff changeset
499 if not fullname or (
bc2535238de2 import-checker: allow relative import a module being checked
Jun Wu <quark@fb.com>
parents: 33938
diff changeset
500 fullname in stdlib_modules
bc2535238de2 import-checker: allow relative import a module being checked
Jun Wu <quark@fb.com>
parents: 33938
diff changeset
501 and fullname not in localmods
bc2535238de2 import-checker: allow relative import a module being checked
Jun Wu <quark@fb.com>
parents: 33938
diff changeset
502 and fullname + '.__init__' not in localmods):
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
503 yield msg('relative import of stdlib module')
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
504 else:
28330
f3fb24e36d61 import-checker: report local with stdlib late warning
timeless <timeless@mozdev.org>
parents: 27621
diff changeset
505 seenlocal = fullname
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
506
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
507 # Direct symbol import is only allowed from certain modules and
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
508 # must occur before non-symbol imports.
29207
a09098c61fea import-checker: always build a list of imported symbols
Yuya Nishihara <yuya@tcha.org>
parents: 29122
diff changeset
509 found = fromlocal(node.module, node.level)
a09098c61fea import-checker: always build a list of imported symbols
Yuya Nishihara <yuya@tcha.org>
parents: 29122
diff changeset
510 if found and found[2]: # node.module is a package
a09098c61fea import-checker: always build a list of imported symbols
Yuya Nishihara <yuya@tcha.org>
parents: 29122
diff changeset
511 prefix = found[0] + '.'
32457
d02888308235 import-checker: add a way to directly import certain symbols
Siddharth Agarwal <sid0@fb.com>
parents: 32413
diff changeset
512 symbols = (n.name for n in node.names
d02888308235 import-checker: add a way to directly import certain symbols
Siddharth Agarwal <sid0@fb.com>
parents: 32413
diff changeset
513 if not fromlocal(prefix + n.name))
29207
a09098c61fea import-checker: always build a list of imported symbols
Yuya Nishihara <yuya@tcha.org>
parents: 29122
diff changeset
514 else:
32457
d02888308235 import-checker: add a way to directly import certain symbols
Siddharth Agarwal <sid0@fb.com>
parents: 32413
diff changeset
515 symbols = (n.name for n in node.names)
d02888308235 import-checker: add a way to directly import certain symbols
Siddharth Agarwal <sid0@fb.com>
parents: 32413
diff changeset
516 symbols = [sym for sym in symbols if sym not in directsymbols]
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
517 if node.module and node.col_offset == root_col_offset:
27272
69308357ecd1 import-checker: allow absolute imports of sub modules from local packages
Yuya Nishihara <yuya@tcha.org>
parents: 27018
diff changeset
518 if symbols and fullname not in allowsymbolimports:
27273
5d5b98346fc2 import-checker: tell which symbol causes "direct symbol import"
Yuya Nishihara <yuya@tcha.org>
parents: 27272
diff changeset
519 yield msg('direct symbol import %s from %s',
5d5b98346fc2 import-checker: tell which symbol causes "direct symbol import"
Yuya Nishihara <yuya@tcha.org>
parents: 27272
diff changeset
520 ', '.join(symbols), fullname)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
521
29208
cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules
Yuya Nishihara <yuya@tcha.org>
parents: 29207
diff changeset
522 if symbols and seennonsymbollocal:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
523 yield msg('symbol import follows non-symbol import: %s',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
524 fullname)
29208
cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules
Yuya Nishihara <yuya@tcha.org>
parents: 29207
diff changeset
525 if not symbols and fullname not in stdlib_modules:
cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules
Yuya Nishihara <yuya@tcha.org>
parents: 29207
diff changeset
526 seennonsymbollocal = True
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
527
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
528 if not node.module:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
529 assert node.level
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
530
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
531 # Only allow 1 group per level.
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
532 if (node.level in seenlevels
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
533 and node.col_offset == root_col_offset):
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
534 yield msg('multiple "from %s import" statements',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
535 '.' * node.level)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
536
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
537 # Higher-level groups come before lower-level groups.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
538 if any(node.level > l for l in seenlevels):
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
539 yield msg('higher-level import should come first: %s',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
540 fullname)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
541
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
542 seenlevels.add(node.level)
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
543
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
544 # Entries in "from .X import ( ... )" lists must be lexically
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
545 # sorted.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
546 lastentryname = None
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
547
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
548 for n in node.names:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
549 if lastentryname and n.name < lastentryname:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
550 yield msg('imports from %s not lexically sorted: %s < %s',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
551 fullname, n.name, lastentryname)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
552
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
553 lastentryname = n.name
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
554
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
555 if n.name in requirealias and n.asname != requirealias[n.name]:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
556 yield msg('%s from %s must be "as" aliased to %s',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
557 n.name, fullname, requirealias[n.name])
25702
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
558
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
559 def verify_stdlib_on_own_line(root):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
560 """Given some python source, verify that stdlib imports are done
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
561 in separate statements from relative local module imports.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
562
25702
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
563 >>> list(verify_stdlib_on_own_line(ast.parse('import sys, foo')))
26956
4b56214ebb7a import-checker: include lineno in warning message
Yuya Nishihara <yuya@tcha.org>
parents: 26955
diff changeset
564 [('mixed imports\\n stdlib: sys\\n relative: foo', 1)]
25702
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
565 >>> list(verify_stdlib_on_own_line(ast.parse('import sys, os')))
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
566 []
25702
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
567 >>> list(verify_stdlib_on_own_line(ast.parse('import foo, bar')))
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
568 []
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
569 """
25702
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
570 for node in ast.walk(root):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
571 if isinstance(node, ast.Import):
20386
a05d31b040d7 import-checker: show stdlib and relative imports separately
Mads Kiilerich <madski@unity3d.com>
parents: 20383
diff changeset
572 from_stdlib = {False: [], True: []}
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
573 for n in node.names:
20386
a05d31b040d7 import-checker: show stdlib and relative imports separately
Mads Kiilerich <madski@unity3d.com>
parents: 20383
diff changeset
574 from_stdlib[n.name in stdlib_modules].append(n.name)
a05d31b040d7 import-checker: show stdlib and relative imports separately
Mads Kiilerich <madski@unity3d.com>
parents: 20383
diff changeset
575 if from_stdlib[True] and from_stdlib[False]:
a05d31b040d7 import-checker: show stdlib and relative imports separately
Mads Kiilerich <madski@unity3d.com>
parents: 20383
diff changeset
576 yield ('mixed imports\n stdlib: %s\n relative: %s' %
a05d31b040d7 import-checker: show stdlib and relative imports separately
Mads Kiilerich <madski@unity3d.com>
parents: 20383
diff changeset
577 (', '.join(sorted(from_stdlib[True])),
26956
4b56214ebb7a import-checker: include lineno in warning message
Yuya Nishihara <yuya@tcha.org>
parents: 26955
diff changeset
578 ', '.join(sorted(from_stdlib[False]))), node.lineno)
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
579
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
580 class CircularImport(Exception):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
581 pass
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
582
24490
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
583 def checkmod(mod, imports):
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
584 shortest = {}
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
585 visit = [[mod]]
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
586 while visit:
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
587 path = visit.pop(0)
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
588 for i in sorted(imports.get(path[-1], [])):
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
589 if len(path) < shortest.get(i, 1000):
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
590 shortest[i] = len(path)
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
591 if i in path:
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
592 if i == path[0]:
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
593 raise CircularImport(path)
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
594 continue
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
595 visit.append(path + [i])
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
596
20038
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
597 def rotatecycle(cycle):
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
598 """arrange a cycle so that the lexicographically first module listed first
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
599
24488
4b3fc46097f7 import-checker: drop duplicate element from cycle
Matt Mackall <mpm@selenic.com>
parents: 24487
diff changeset
600 >>> rotatecycle(['foo', 'bar'])
20038
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
601 ['bar', 'foo', 'bar']
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
602 """
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
603 lowest = min(cycle)
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
604 idx = cycle.index(lowest)
24488
4b3fc46097f7 import-checker: drop duplicate element from cycle
Matt Mackall <mpm@selenic.com>
parents: 24487
diff changeset
605 return cycle[idx:] + cycle[:idx] + [lowest]
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
606
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
607 def find_cycles(imports):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
608 """Find cycles in an already-loaded import graph.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
609
25175
10e6c4b7121b import-checker: don't treat modules as relative one if not found
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25174
diff changeset
610 All module names recorded in `imports` should be absolute one.
10e6c4b7121b import-checker: don't treat modules as relative one if not found
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25174
diff changeset
611
28703
a274c4f9087a py3: use print_function in import-checker
timeless <timeless@mozdev.org>
parents: 28702
diff changeset
612 >>> from __future__ import print_function
25175
10e6c4b7121b import-checker: don't treat modules as relative one if not found
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25174
diff changeset
613 >>> imports = {'top.foo': ['top.bar', 'os.path', 'top.qux'],
10e6c4b7121b import-checker: don't treat modules as relative one if not found
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25174
diff changeset
614 ... 'top.bar': ['top.baz', 'sys'],
10e6c4b7121b import-checker: don't treat modules as relative one if not found
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25174
diff changeset
615 ... 'top.baz': ['top.foo'],
10e6c4b7121b import-checker: don't treat modules as relative one if not found
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25174
diff changeset
616 ... 'top.qux': ['top.foo']}
28703
a274c4f9087a py3: use print_function in import-checker
timeless <timeless@mozdev.org>
parents: 28702
diff changeset
617 >>> print('\\n'.join(sorted(find_cycles(imports))))
24487
642d245ff537 import-checker: fix rotatecycle
Matt Mackall <mpm@selenic.com>
parents: 22975
diff changeset
618 top.bar -> top.baz -> top.foo -> top.bar
642d245ff537 import-checker: fix rotatecycle
Matt Mackall <mpm@selenic.com>
parents: 22975
diff changeset
619 top.foo -> top.qux -> top.foo
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
620 """
24491
784b278b349c import-checker: rotatecycle is actually the canonical cycle key
Matt Mackall <mpm@selenic.com>
parents: 24490
diff changeset
621 cycles = set()
28704
1fa6fdb72275 py3: handle iter/iterkeys+iteritems python3 divergence in import-checker
timeless <timeless@mozdev.org>
parents: 28703
diff changeset
622 for mod in sorted(imports.keys()):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
623 try:
24490
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
624 checkmod(mod, imports)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25175
diff changeset
625 except CircularImport as e:
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
626 cycle = e.args[0]
24491
784b278b349c import-checker: rotatecycle is actually the canonical cycle key
Matt Mackall <mpm@selenic.com>
parents: 24490
diff changeset
627 cycles.add(" -> ".join(rotatecycle(cycle)))
784b278b349c import-checker: rotatecycle is actually the canonical cycle key
Matt Mackall <mpm@selenic.com>
parents: 24490
diff changeset
628 return cycles
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
629
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
630 def _cycle_sortkey(c):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
631 return len(c), c
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
632
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
633 def embedded(f, modname, src):
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
634 """Extract embedded python code
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
635
33896
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
636 >>> def _forcestr(thing):
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
637 ... if not isinstance(thing, str):
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
638 ... return thing.decode('ascii')
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
639 ... return thing
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
640 >>> def test(fn, lines):
33896
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
641 ... for s, m, f, l in embedded(fn, b"example", lines):
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
642 ... print("%s %s %d" % (_forcestr(m), _forcestr(f), l))
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
643 ... print(repr(_forcestr(s)))
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
644 >>> lines = [
33896
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
645 ... b'comment',
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
646 ... b' >>> from __future__ import print_function',
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
647 ... b" >>> ' multiline",
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
648 ... b" ... string'",
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
649 ... b' ',
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
650 ... b'comment',
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
651 ... b' $ cat > foo.py <<EOF',
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
652 ... b' > from __future__ import print_function',
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
653 ... b' > EOF',
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
654 ... ]
33896
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
655 >>> test(b"example.t", lines)
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
656 example[2] doctest.py 2
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
657 "from __future__ import print_function\\n' multiline\\nstring'\\n"
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
658 example[7] foo.py 7
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
659 'from __future__ import print_function\\n'
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
660 """
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
661 inlinepython = 0
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
662 shpython = 0
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
663 script = []
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
664 prefix = 6
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
665 t = ''
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
666 n = 0
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
667 for l in src:
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
668 n += 1
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
669 if not l.endswith(b'\n'):
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
670 l += b'\n'
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
671 if l.startswith(b' >>> '): # python inlines
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
672 if shpython:
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
673 print("%s:%d: Parse Error" % (f, n))
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
674 if not inlinepython:
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
675 # We've just entered a Python block.
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
676 inlinepython = n
33896
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
677 t = b'doctest.py'
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
678 script.append(l[prefix:])
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
679 continue
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
680 if l.startswith(b' ... '): # python inlines
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
681 script.append(l[prefix:])
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
682 continue
33896
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
683 cat = re.search(br"\$ \s*cat\s*>\s*(\S+\.py)\s*<<\s*EOF", l)
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
684 if cat:
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
685 if inlinepython:
33896
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
686 yield b''.join(script), (b"%s[%d]" %
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
687 (modname, inlinepython)), t, inlinepython
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
688 script = []
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
689 inlinepython = 0
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
690 shpython = n
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
691 t = cat.group(1)
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
692 continue
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
693 if shpython and l.startswith(b' > '): # sh continuation
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
694 if l == b' > EOF\n':
33896
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
695 yield b''.join(script), (b"%s[%d]" %
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
696 (modname, shpython)), t, shpython
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
697 script = []
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
698 shpython = 0
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
699 else:
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
700 script.append(l[4:])
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
701 continue
33938
8a8dd6e4a97a contrib: make import-checker agree more with run-tests about heredocpy
Augie Fackler <raf@durin42.com>
parents: 33937
diff changeset
702 # If we have an empty line or a command for sh, we end the
8a8dd6e4a97a contrib: make import-checker agree more with run-tests about heredocpy
Augie Fackler <raf@durin42.com>
parents: 33937
diff changeset
703 # inline script.
8a8dd6e4a97a contrib: make import-checker agree more with run-tests about heredocpy
Augie Fackler <raf@durin42.com>
parents: 33937
diff changeset
704 if inlinepython and (l == b' \n'
8a8dd6e4a97a contrib: make import-checker agree more with run-tests about heredocpy
Augie Fackler <raf@durin42.com>
parents: 33937
diff changeset
705 or l.startswith(b' $ ')):
33896
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
706 yield b''.join(script), (b"%s[%d]" %
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
707 (modname, inlinepython)), t, inlinepython
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
708 script = []
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
709 inlinepython = 0
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
710 continue
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
711
28919
a94f34306bb9 import-checker: refactor source reading
timeless <timeless@mozdev.org>
parents: 28713
diff changeset
712 def sources(f, modname):
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
713 """Yields possibly multiple sources from a filepath
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
714
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
715 input: filepath, modulename
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
716 yields: script(string), modulename, filepath, linenumber
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
717
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
718 For embedded scripts, the modulename and filepath will be different
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
719 from the function arguments. linenumber is an offset relative to
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
720 the input file.
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
721 """
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
722 py = False
29234
393aef802535 tests: enable import checker for all python files (including no .py files)
Yuya Nishihara <yuya@tcha.org>
parents: 29211
diff changeset
723 if not f.endswith('.t'):
33896
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
724 with open(f, 'rb') as src:
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
725 yield src.read(), modname, f, 0
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
726 py = True
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
727 if py or f.endswith('.t'):
33896
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33895
diff changeset
728 with open(f, 'rb') as src:
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
729 for script, modname, t, line in embedded(f, modname, src):
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
730 yield script, modname, t, line
28919
a94f34306bb9 import-checker: refactor source reading
timeless <timeless@mozdev.org>
parents: 28713
diff changeset
731
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
732 def main(argv):
25063
723e364488f4 import-checker: add xargs like mode
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24669
diff changeset
733 if len(argv) < 2 or (argv[1] == '-' and len(argv) > 2):
28703
a274c4f9087a py3: use print_function in import-checker
timeless <timeless@mozdev.org>
parents: 28702
diff changeset
734 print('Usage: %s {-|file [file] [file] ...}')
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
735 return 1
25063
723e364488f4 import-checker: add xargs like mode
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24669
diff changeset
736 if argv[1] == '-':
723e364488f4 import-checker: add xargs like mode
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24669
diff changeset
737 argv = argv[:1]
723e364488f4 import-checker: add xargs like mode
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24669
diff changeset
738 argv.extend(l.rstrip() for l in sys.stdin.readlines())
32541
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32540
diff changeset
739 localmodpaths = {}
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
740 used_imports = {}
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
741 any_errors = False
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
742 for source_path in argv[1:]:
32413
194b0f781132 import-checker: drop workaround for pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32411
diff changeset
743 modname = dotted_name_of_path(source_path)
32541
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32540
diff changeset
744 localmodpaths[modname] = source_path
32542
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32541
diff changeset
745 localmods = populateextmods(localmodpaths)
32541
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32540
diff changeset
746 for localmodname, source_path in sorted(localmodpaths.items()):
33908
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names
Augie Fackler <raf@durin42.com>
parents: 33896
diff changeset
747 if not isinstance(localmodname, bytes):
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names
Augie Fackler <raf@durin42.com>
parents: 33896
diff changeset
748 # This is only safe because all hg's files are ascii
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names
Augie Fackler <raf@durin42.com>
parents: 33896
diff changeset
749 localmodname = localmodname.encode('ascii')
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
750 for src, modname, name, line in sources(source_path, localmodname):
28920
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
751 try:
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
752 used_imports[modname] = sorted(
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
753 imported_modules(src, modname, name, localmods,
28920
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
754 ignore_nested=True))
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
755 for error, lineno in verify_import_convention(modname, src,
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
756 localmods):
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
757 any_errors = True
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
758 print('%s:%d: %s' % (source_path, lineno + line, error))
28920
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
759 except SyntaxError as e:
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
760 print('%s:%d: SyntaxError: %s' %
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
761 (source_path, e.lineno + line, e))
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
762 cycles = find_cycles(used_imports)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
763 if cycles:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
764 firstmods = set()
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
765 for c in sorted(cycles, key=_cycle_sortkey):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
766 first = c.split()[0]
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
767 # As a rough cut, ignore any cycle that starts with the
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
768 # same module as some other cycle. Otherwise we see lots
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
769 # of cycles that are effectively duplicates.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
770 if first in firstmods:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
771 continue
28703
a274c4f9087a py3: use print_function in import-checker
timeless <timeless@mozdev.org>
parents: 28702
diff changeset
772 print('Import cycle:', c)
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
773 firstmods.add(first)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
774 any_errors = True
25731
cd1daab5d036 import-checker.py: exit with code 0 if no error is detected
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25703
diff changeset
775 return any_errors != 0
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
776
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
777 if __name__ == '__main__':
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
778 sys.exit(int(main(sys.argv)))