annotate contrib/import-checker.py @ 41426:02d0a7774882

py3: byteify the LFS blobstore module This is almost entirely b'' prefixing, with a couple of exceptions forced to bytes. Much of this is also borrowed from Augie's code. There's an HTTPError.read() that I flagged that I assume needs to be converted to bytes, but I can't find confirmation. Handling the deserialized JSON object over several functions made r'' vs b'' accesses confusing, so this assumes that the JSON object will be converted to bytes immediately. That will be done in the following commits, so it's not buried in these trivial changes.
author Matt Harbison <matt_harbison@yahoo.com>
date Sun, 27 Jan 2019 15:19:28 -0500
parents 98ce494e104d
children 04eb3c5607af
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
26954
f804bf27439b import-checker: make it executable for convenience
Yuya Nishihara <yuya@tcha.org>
parents: 26781
diff changeset
1 #!/usr/bin/env python
f804bf27439b import-checker: make it executable for convenience
Yuya Nishihara <yuya@tcha.org>
parents: 26781
diff changeset
2
28703
a274c4f9087a py3: use print_function in import-checker
timeless <timeless@mozdev.org>
parents: 28702
diff changeset
3 from __future__ import absolute_import, print_function
28702
e44f671018e3 py3: use absolute_import in import-checker
timeless <timeless@mozdev.org>
parents: 28700
diff changeset
4
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
5 import ast
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
6 import collections
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
7 import os
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
8 import sys
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
9
20198
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
10 # Import a minimal set of stdlib modules needed for list_stdlib_modules()
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
11 # to work when run from a virtualenv. The modules were chosen empirically
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
12 # so that the return value matches the return value without virtualenv.
29211
b42c2a66a698 py3: make contrib/import-checker.py get along with itself
Yuya Nishihara <yuya@tcha.org>
parents: 29208
diff changeset
13 if True: # disable lexical sorting checks
33876
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32595
diff changeset
14 try:
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32595
diff changeset
15 import BaseHTTPServer as basehttpserver
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32595
diff changeset
16 except ImportError:
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32595
diff changeset
17 basehttpserver = None
29211
b42c2a66a698 py3: make contrib/import-checker.py get along with itself
Yuya Nishihara <yuya@tcha.org>
parents: 29208
diff changeset
18 import zlib
20198
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
19
40095
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 39329
diff changeset
20 import testparseutil
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 39329
diff changeset
21
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
22 # Whitelist of modules that symbols can be directly imported from.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
23 allowsymbolimports = (
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
24 '__future__',
33919
5ed0be4d9df9 contrib: add bzrlib to list of packages from which we import symbols
Augie Fackler <raf@durin42.com>
parents: 33915
diff changeset
25 'bzrlib',
33915
2d64b2f1787b contrib: allow symbol imports from hgclient for tests
Augie Fackler <raf@durin42.com>
parents: 33897
diff changeset
26 'hgclient',
33893
c9cf69d0c3b9 contrib: allow importing "symbols" from mercurial
Augie Fackler <raf@durin42.com>
parents: 33890
diff changeset
27 'mercurial',
27018
e5be48dd8215 import-checker: allow symbol imports from hgweb.common and .request
Yuya Nishihara <yuya@tcha.org>
parents: 26965
diff changeset
28 'mercurial.hgweb.common',
e5be48dd8215 import-checker: allow symbol imports from hgweb.common and .request
Yuya Nishihara <yuya@tcha.org>
parents: 26965
diff changeset
29 'mercurial.hgweb.request',
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
30 'mercurial.i18n',
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
31 'mercurial.node',
39329
729082bb9938 revlog: split constants into a new `revlogutils.constants` module
Boris Feld <boris.feld@octobus.net>
parents: 38797
diff changeset
32 # for revlog to re-export constant to extensions
729082bb9938 revlog: split constants into a new `revlogutils.constants` module
Boris Feld <boris.feld@octobus.net>
parents: 38797
diff changeset
33 'mercurial.revlogutils.constants',
32507
95085d747db8 import-checker: allow importing symbols from pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32420
diff changeset
34 # for cffi modules to re-export pure functions
95085d747db8 import-checker: allow importing symbols from pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32420
diff changeset
35 'mercurial.pure.base85',
95085d747db8 import-checker: allow importing symbols from pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32420
diff changeset
36 'mercurial.pure.bdiff',
95085d747db8 import-checker: allow importing symbols from pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32420
diff changeset
37 'mercurial.pure.mpatch',
95085d747db8 import-checker: allow importing symbols from pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32420
diff changeset
38 'mercurial.pure.osutil',
95085d747db8 import-checker: allow importing symbols from pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32420
diff changeset
39 'mercurial.pure.parsers',
34395
41401f502c83 tests: disable lints on mercurial/thirdparty
Siddharth Agarwal <sid0@fb.com>
parents: 34038
diff changeset
40 # third-party imports should be directly imported
41401f502c83 tests: disable lints on mercurial/thirdparty
Siddharth Agarwal <sid0@fb.com>
parents: 34038
diff changeset
41 'mercurial.thirdparty',
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 37711
diff changeset
42 'mercurial.thirdparty.attr',
37180
922b3fae9c7d setup: register zope.interface packages and compile C extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34395
diff changeset
43 'mercurial.thirdparty.zope',
922b3fae9c7d setup: register zope.interface packages and compile C extension
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34395
diff changeset
44 'mercurial.thirdparty.zope.interface',
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
45 )
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
46
32419
d02888308235 import-checker: add a way to directly import certain symbols
Siddharth Agarwal <sid0@fb.com>
parents: 32374
diff changeset
47 # Whitelist of symbols that can be directly imported.
32420
0906b85bf222 demandimport: move to separate package
Siddharth Agarwal <sid0@fb.com>
parents: 32419
diff changeset
48 directsymbols = (
0906b85bf222 demandimport: move to separate package
Siddharth Agarwal <sid0@fb.com>
parents: 32419
diff changeset
49 'demandimport',
0906b85bf222 demandimport: move to separate package
Siddharth Agarwal <sid0@fb.com>
parents: 32419
diff changeset
50 )
32419
d02888308235 import-checker: add a way to directly import certain symbols
Siddharth Agarwal <sid0@fb.com>
parents: 32374
diff changeset
51
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
52 # Modules that must be aliased because they are commonly confused with
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
53 # common variables and can create aliasing and readability issues.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
54 requirealias = {
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
55 'ui': 'uimod',
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
56 }
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
57
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
58 def usingabsolute(root):
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
59 """Whether absolute imports are being used."""
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
60 if sys.version_info[0] >= 3:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
61 return True
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
62
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
63 for node in ast.walk(root):
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
64 if isinstance(node, ast.ImportFrom):
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
65 if node.module == '__future__':
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
66 for n in node.names:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
67 if n.name == 'absolute_import':
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
68 return True
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
69
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
70 return False
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
71
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
72 def walklocal(root):
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
73 """Recursively yield all descendant nodes but not in a different scope"""
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
74 todo = collections.deque(ast.iter_child_nodes(root))
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
75 yield root, False
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
76 while todo:
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
77 node = todo.popleft()
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
78 newscope = isinstance(node, ast.FunctionDef)
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
79 if not newscope:
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
80 todo.extend(ast.iter_child_nodes(node))
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
81 yield node, newscope
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
82
32374
194b0f781132 import-checker: drop workaround for pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32372
diff changeset
83 def dotted_name_of_path(path):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
84 """Given a relative path to a source file, return its dotted module name.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
85
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
86 >>> dotted_name_of_path('mercurial/error.py')
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
87 'mercurial.error'
20383
4990abb4729d import-checker: fix names of dynamically loaded modules
Mads Kiilerich <madski@unity3d.com>
parents: 20238
diff changeset
88 >>> dotted_name_of_path('zlibmodule.so')
4990abb4729d import-checker: fix names of dynamically loaded modules
Mads Kiilerich <madski@unity3d.com>
parents: 20238
diff changeset
89 'zlib'
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
90 """
27620
0c60843b55b5 import-checker: normalize directory separator to get module name on Windows
Yuya Nishihara <yuya@tcha.org>
parents: 27520
diff changeset
91 parts = path.replace(os.sep, '/').split('/')
20391
466e4c574db0 import-checker: handle standard modules with arch in the filename
Mads Kiilerich <madski@unity3d.com>
parents: 20386
diff changeset
92 parts[-1] = parts[-1].split('.', 1)[0] # remove .py and .so and .ARCH.so
20383
4990abb4729d import-checker: fix names of dynamically loaded modules
Mads Kiilerich <madski@unity3d.com>
parents: 20238
diff changeset
93 if parts[-1].endswith('module'):
4990abb4729d import-checker: fix names of dynamically loaded modules
Mads Kiilerich <madski@unity3d.com>
parents: 20238
diff changeset
94 parts[-1] = parts[-1][:-6]
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
95 return '.'.join(parts)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
96
25173
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
97 def fromlocalfunc(modulename, localmods):
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
98 """Get a function to examine which locally defined module the
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
99 target source imports via a specified name.
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
100
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
101 `modulename` is an `dotted_name_of_path()`-ed source file path,
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
102 which may have `.__init__` at the end of it, of the target source.
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
103
32508
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32507
diff changeset
104 `localmods` is a set of absolute `dotted_name_of_path()`-ed source file
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32507
diff changeset
105 paths of locally defined (= Mercurial specific) modules.
25173
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
106
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
107 This function assumes that module names not existing in
26781
1aee2ab0f902 spelling: trivial spell checking
Mads Kiilerich <madski@unity3d.com>
parents: 26221
diff changeset
108 `localmods` are from the Python standard library.
25173
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
109
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
110 This function returns the function, which takes `name` argument,
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
111 and returns `(absname, dottedpath, hassubmod)` tuple if `name`
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
112 matches against locally defined module. Otherwise, it returns
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
113 False.
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
114
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
115 It is assumed that `name` doesn't have `.__init__`.
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
116
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
117 `absname` is an absolute module name of specified `name`
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
118 (e.g. "hgext.convert"). This can be used to compose prefix for sub
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
119 modules or so.
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
120
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
121 `dottedpath` is a `dotted_name_of_path()`-ed source file path
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
122 (e.g. "hgext.convert.__init__") of `name`. This is used to look
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
123 module up in `localmods` again.
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
124
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
125 `hassubmod` is whether it may have sub modules under it (for
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
126 convenient, even though this is also equivalent to "absname !=
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
127 dottednpath")
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
128
32508
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32507
diff changeset
129 >>> localmods = {'foo.__init__', 'foo.foo1',
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32507
diff changeset
130 ... 'foo.bar.__init__', 'foo.bar.bar1',
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32507
diff changeset
131 ... 'baz.__init__', 'baz.baz1'}
25173
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
132 >>> fromlocal = fromlocalfunc('foo.xxx', localmods)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
133 >>> # relative
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
134 >>> fromlocal('foo1')
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
135 ('foo.foo1', 'foo.foo1', False)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
136 >>> fromlocal('bar')
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
137 ('foo.bar', 'foo.bar.__init__', True)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
138 >>> fromlocal('bar.bar1')
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
139 ('foo.bar.bar1', 'foo.bar.bar1', False)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
140 >>> # absolute
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
141 >>> fromlocal('baz')
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
142 ('baz', 'baz.__init__', True)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
143 >>> fromlocal('baz.baz1')
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
144 ('baz.baz1', 'baz.baz1', False)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
145 >>> # unknown = maybe standard library
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
146 >>> fromlocal('os')
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
147 False
25701
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
148 >>> fromlocal(None, 1)
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
149 ('foo', 'foo.__init__', True)
29122
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
150 >>> fromlocal('foo1', 1)
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
151 ('foo.foo1', 'foo.foo1', False)
25701
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
152 >>> fromlocal2 = fromlocalfunc('foo.xxx.yyy', localmods)
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
153 >>> fromlocal2(None, 2)
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
154 ('foo', 'foo.__init__', True)
29122
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
155 >>> fromlocal2('bar2', 1)
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
156 False
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
157 >>> fromlocal2('bar', 2)
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
158 ('foo.bar', 'foo.bar.__init__', True)
25173
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
159 """
33890
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names
Augie Fackler <raf@durin42.com>
parents: 33877
diff changeset
160 if not isinstance(modulename, str):
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names
Augie Fackler <raf@durin42.com>
parents: 33877
diff changeset
161 modulename = modulename.decode('ascii')
25173
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
162 prefix = '.'.join(modulename.split('.')[:-1])
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
163 if prefix:
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
164 prefix += '.'
25701
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
165 def fromlocal(name, level=0):
29374
7712fcde2d56 import-checker: increase portability for python 2.6.x
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29234
diff changeset
166 # name is false value when relative imports are used.
7712fcde2d56 import-checker: increase portability for python 2.6.x
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29234
diff changeset
167 if not name:
25701
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
168 # If relative imports are used, level must not be absolute.
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
169 assert level > 0
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
170 candidates = ['.'.join(modulename.split('.')[:-level])]
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
171 else:
29122
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
172 if not level:
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
173 # Check relative name first.
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
174 candidates = [prefix + name, name]
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
175 else:
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
176 candidates = ['.'.join(modulename.split('.')[:-level]) +
660d8d4ec7aa import-checker: recognize relative imports from parents of current package
liscju <piotr.listkiewicz@gmail.com>
parents: 28922
diff changeset
177 '.' + name]
25701
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
178
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
179 for n in candidates:
25173
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
180 if n in localmods:
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
181 return (n, n, False)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
182 dottedpath = n + '.__init__'
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
183 if dottedpath in localmods:
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
184 return (n, dottedpath, True)
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
185 return False
7358b5d9991e import-checker: add utility to examine what module is imported easily
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25064
diff changeset
186 return fromlocal
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
187
32509
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32508
diff changeset
188 def populateextmods(localmods):
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32508
diff changeset
189 """Populate C extension modules based on pure modules"""
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32508
diff changeset
190 newlocalmods = set(localmods)
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32508
diff changeset
191 for n in localmods:
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32508
diff changeset
192 if n.startswith('mercurial.pure.'):
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32508
diff changeset
193 m = n[len('mercurial.pure.'):]
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32508
diff changeset
194 newlocalmods.add('mercurial.cext.' + m)
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32508
diff changeset
195 newlocalmods.add('mercurial.cffi._' + m)
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32508
diff changeset
196 return newlocalmods
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32508
diff changeset
197
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
198 def list_stdlib_modules():
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
199 """List the modules present in the stdlib.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
200
33876
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32595
diff changeset
201 >>> py3 = sys.version_info[0] >= 3
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
202 >>> mods = set(list_stdlib_modules())
33876
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32595
diff changeset
203 >>> 'BaseHTTPServer' in mods or py3
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
204 True
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
205
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
206 os.path isn't really a module, so it's missing:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
207
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
208 >>> 'os.path' in mods
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
209 False
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
210
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
211 sys requires special treatment, because it's baked into the
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
212 interpreter, but it should still appear:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
213
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
214 >>> 'sys' in mods
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
215 True
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
216
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
217 >>> 'collections' in mods
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
218 True
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
219
33876
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32595
diff changeset
220 >>> 'cStringIO' in mods or py3
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
221 True
29395
4c8026babe8c import-checker: ensure cffi is always a system module
Augie Fackler <raf@durin42.com>
parents: 29374
diff changeset
222
4c8026babe8c import-checker: ensure cffi is always a system module
Augie Fackler <raf@durin42.com>
parents: 29374
diff changeset
223 >>> 'cffi' in mods
4c8026babe8c import-checker: ensure cffi is always a system module
Augie Fackler <raf@durin42.com>
parents: 29374
diff changeset
224 True
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
225 """
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
226 for m in sys.builtin_module_names:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
227 yield m
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
228 # These modules only exist on windows, but we should always
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
229 # consider them stdlib.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
230 for m in ['msvcrt', '_winreg']:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
231 yield m
33894
c856cb1c29be contrib: inform import checker that __builtin__ is a thing
Augie Fackler <raf@durin42.com>
parents: 33893
diff changeset
232 yield '__builtin__'
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
233 yield 'builtins' # python3 only
33897
b4294ff13392 contrib: always treat importlib.* as stdlib
Augie Fackler <raf@durin42.com>
parents: 33894
diff changeset
234 yield 'importlib.abc' # python3 only
b4294ff13392 contrib: always treat importlib.* as stdlib
Augie Fackler <raf@durin42.com>
parents: 33894
diff changeset
235 yield 'importlib.machinery' # python3 only
b4294ff13392 contrib: always treat importlib.* as stdlib
Augie Fackler <raf@durin42.com>
parents: 33894
diff changeset
236 yield 'importlib.util' # python3 only
24669
fbdbff1b486a import-checker: force 'fcntl', 'grp', 'pwd', and 'termios' to stdlib modules
Matt Harbison <matt_harbison@yahoo.com>
parents: 24668
diff changeset
237 for m in 'fcntl', 'grp', 'pwd', 'termios': # Unix only
fbdbff1b486a import-checker: force 'fcntl', 'grp', 'pwd', and 'termios' to stdlib modules
Matt Harbison <matt_harbison@yahoo.com>
parents: 24668
diff changeset
238 yield m
28713
806d260c6f3b tests: fix builtin module test on pypy
Maciej Fijalkowski <fijall@gmail.com>
parents: 28704
diff changeset
239 for m in 'cPickle', 'datetime': # in Python (not C) on PyPy
806d260c6f3b tests: fix builtin module test on pypy
Maciej Fijalkowski <fijall@gmail.com>
parents: 28704
diff changeset
240 yield m
29395
4c8026babe8c import-checker: ensure cffi is always a system module
Augie Fackler <raf@durin42.com>
parents: 29374
diff changeset
241 for m in ['cffi']:
4c8026babe8c import-checker: ensure cffi is always a system module
Augie Fackler <raf@durin42.com>
parents: 29374
diff changeset
242 yield m
32291
bd872f64a8ba cleanup: use set literals
Martin von Zweigbergk <martinvonz@google.com>
parents: 32212
diff changeset
243 stdlib_prefixes = {sys.prefix, sys.exec_prefix}
20198
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
244 # We need to supplement the list of prefixes for the search to work
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
245 # when run from within a virtualenv.
33876
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32595
diff changeset
246 for mod in (basehttpserver, zlib):
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32595
diff changeset
247 if mod is None:
998fad4b3072 contrib: work around some modules not existing on Py3 in import checker
Augie Fackler <raf@durin42.com>
parents: 32595
diff changeset
248 continue
20198
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
249 try:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
250 # Not all module objects have a __file__ attribute.
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
251 filename = mod.__file__
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
252 except AttributeError:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
253 continue
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
254 dirname = os.path.dirname(filename)
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
255 for prefix in stdlib_prefixes:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
256 if dirname.startswith(prefix):
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
257 # Then this directory is redundant.
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
258 break
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
259 else:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
260 stdlib_prefixes.add(dirname)
40687
dd028bca9221 tests: make test-check-module-imports more robust
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 40481
diff changeset
261 sourceroot = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
262 for libpath in sys.path:
40687
dd028bca9221 tests: make test-check-module-imports more robust
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 40481
diff changeset
263 # We want to walk everything in sys.path that starts with something in
dd028bca9221 tests: make test-check-module-imports more robust
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 40481
diff changeset
264 # stdlib_prefixes, but not directories from the hg sources.
dd028bca9221 tests: make test-check-module-imports more robust
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 40481
diff changeset
265 if (os.path.abspath(libpath).startswith(sourceroot)
dd028bca9221 tests: make test-check-module-imports more robust
Valentin Gatien-Baron <vgatien-baron@janestreet.com>
parents: 40481
diff changeset
266 or not any(libpath.startswith(p) for p in stdlib_prefixes)):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
267 continue
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
268 for top, dirs, files in os.walk(libpath):
25733
f99c066f5f9a import-checker: recurse into subtree of sys.path only if __init__.py exists
Yuya Nishihara <yuya@tcha.org>
parents: 25731
diff changeset
269 for i, d in reversed(list(enumerate(dirs))):
25734
9086d0c1def3 import-checker: exclude mercurial packages installed into the system path
Yuya Nishihara <yuya@tcha.org>
parents: 25733
diff changeset
270 if (not os.path.exists(os.path.join(top, d, '__init__.py'))
32595
9e46627baa3c import-checker: add hgdemandimport to local modules
Siddharth Agarwal <sid0@fb.com>
parents: 32509
diff changeset
271 or top == libpath and d in ('hgdemandimport', 'hgext',
9e46627baa3c import-checker: add hgdemandimport to local modules
Siddharth Agarwal <sid0@fb.com>
parents: 32509
diff changeset
272 'mercurial')):
25733
f99c066f5f9a import-checker: recurse into subtree of sys.path only if __init__.py exists
Yuya Nishihara <yuya@tcha.org>
parents: 25731
diff changeset
273 del dirs[i]
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
274 for name in files:
26221
ae65b1b4cb46 import-checker: use modern .endswith for multiple suffixes
Augie Fackler <augie@google.com>
parents: 26166
diff changeset
275 if not name.endswith(('.py', '.so', '.pyc', '.pyo', '.pyd')):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
276 continue
27621
39845b064041 import-checker: list package directory as stdlib module
Yuya Nishihara <yuya@tcha.org>
parents: 27620
diff changeset
277 if name.startswith('__init__.py'):
39845b064041 import-checker: list package directory as stdlib module
Yuya Nishihara <yuya@tcha.org>
parents: 27620
diff changeset
278 full_path = top
39845b064041 import-checker: list package directory as stdlib module
Yuya Nishihara <yuya@tcha.org>
parents: 27620
diff changeset
279 else:
39845b064041 import-checker: list package directory as stdlib module
Yuya Nishihara <yuya@tcha.org>
parents: 27620
diff changeset
280 full_path = os.path.join(top, name)
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
281 rel_path = full_path[len(libpath) + 1:]
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
282 mod = dotted_name_of_path(rel_path)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
283 yield mod
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
284
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
285 stdlib_modules = set(list_stdlib_modules())
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
286
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
287 def imported_modules(source, modulename, f, localmods, ignore_nested=False):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
288 """Given the source of a file as a string, yield the names
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
289 imported by that file.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
290
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
291 Args:
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
292 source: The python source to examine as a string.
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
293 modulename: of specified python source (may have `__init__`)
32508
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32507
diff changeset
294 localmods: set of locally defined module names (may have `__init__`)
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
295 ignore_nested: If true, import statements that do not start in
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
296 column zero will be ignored.
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
297
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
298 Returns:
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
299 A list of absolute module names imported by the given source.
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
300
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
301 >>> f = 'foo/xxx.py'
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
302 >>> modulename = 'foo.xxx'
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
303 >>> localmods = {'foo.__init__': True,
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
304 ... 'foo.foo1': True, 'foo.foo2': True,
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
305 ... 'foo.bar.__init__': True, 'foo.bar.bar1': True,
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
306 ... 'baz.__init__': True, 'baz.baz1': True }
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
307 >>> # standard library (= not locally defined ones)
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
308 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
309 ... 'from stdlib1 import foo, bar; import stdlib2',
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
310 ... modulename, f, localmods))
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
311 []
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
312 >>> # relative importing
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
313 >>> sorted(imported_modules(
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
314 ... 'import foo1; from bar import bar1',
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
315 ... modulename, f, localmods))
26964
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
316 ['foo.bar.bar1', 'foo.foo1']
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
317 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
318 ... 'from bar.bar1 import name1, name2, name3',
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
319 ... modulename, f, localmods))
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
320 ['foo.bar.bar1']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
321 >>> # absolute importing
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
322 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
323 ... 'from baz import baz1, name1',
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
324 ... modulename, f, localmods))
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
325 ['baz.__init__', 'baz.baz1']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
326 >>> # mixed importing, even though it shouldn't be recommended
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
327 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
328 ... 'import stdlib, foo1, baz',
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
329 ... modulename, f, localmods))
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
330 ['baz.__init__', 'foo.foo1']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
331 >>> # ignore_nested
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
332 >>> sorted(imported_modules(
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
333 ... '''import foo
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
334 ... def wat():
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
335 ... import bar
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
336 ... ''', modulename, f, localmods))
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
337 ['foo.__init__', 'foo.bar.__init__']
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
338 >>> sorted(imported_modules(
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
339 ... '''import foo
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
340 ... def wat():
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
341 ... import bar
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
342 ... ''', modulename, f, localmods, ignore_nested=True))
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
343 ['foo.__init__']
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
344 """
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
345 fromlocal = fromlocalfunc(modulename, localmods)
28921
02ee31a50002 import-checker: track filenames for SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28920
diff changeset
346 for node in ast.walk(ast.parse(source, f)):
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
347 if ignore_nested and getattr(node, 'col_offset', 0) > 0:
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
348 continue
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
349 if isinstance(node, ast.Import):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
350 for n in node.names:
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
351 found = fromlocal(n.name)
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
352 if not found:
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
353 # this should import standard library
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
354 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
355 yield found[1]
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
356 elif isinstance(node, ast.ImportFrom):
25701
1f88c0f6ff5a import-checker: resolve relative imports
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25660
diff changeset
357 found = fromlocal(node.module, node.level)
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
358 if not found:
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
359 # this should import standard library
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
360 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
361
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
362 absname, dottedpath, hassubmod = found
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
363 if not hassubmod:
26964
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
364 # "dottedpath" is not a package; must be imported
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
365 yield dottedpath
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
366 # examination of "node.names" should be redundant
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
367 # e.g.: from mercurial.node import nullid, nullrev
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
368 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
369
26964
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
370 modnotfound = False
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
371 prefix = absname + '.'
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
372 for n in node.names:
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
373 found = fromlocal(prefix + n.name)
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
374 if not found:
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
375 # this should be a function or a property of "node.module"
26964
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
376 modnotfound = True
25174
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
377 continue
86298718b01c import-checker: make imported_modules yield absolute dotted_name_of_path
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25173
diff changeset
378 yield found[1]
26964
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
379 if modnotfound:
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
380 # "dottedpath" is a package, but imported because of non-module
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
381 # lookup
5abba2c92da3 import-checker: allow import of child modules from package root
Yuya Nishihara <yuya@tcha.org>
parents: 26956
diff changeset
382 yield dottedpath
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
383
27272
69308357ecd1 import-checker: allow absolute imports of sub modules from local packages
Yuya Nishihara <yuya@tcha.org>
parents: 27018
diff changeset
384 def verify_import_convention(module, source, localmods):
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
385 """Verify imports match our established coding convention.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
386
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
387 We have 2 conventions: legacy and modern. The modern convention is in
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
388 effect when using absolute imports.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
389
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
390 The legacy convention only looks for mixed imports. The modern convention
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
391 is much more thorough.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
392 """
25702
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
393 root = ast.parse(source)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
394 absolute = usingabsolute(root)
25702
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
395
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
396 if absolute:
27272
69308357ecd1 import-checker: allow absolute imports of sub modules from local packages
Yuya Nishihara <yuya@tcha.org>
parents: 27018
diff changeset
397 return verify_modern_convention(module, root, localmods)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
398 else:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
399 return verify_stdlib_on_own_line(root)
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
400
27272
69308357ecd1 import-checker: allow absolute imports of sub modules from local packages
Yuya Nishihara <yuya@tcha.org>
parents: 27018
diff changeset
401 def verify_modern_convention(module, root, localmods, root_col_offset=0):
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
402 """Verify a file conforms to the modern import convention rules.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
403
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
404 The rules of the modern convention are:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
405
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
406 * Ordering is stdlib followed by local imports. Each group is lexically
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
407 sorted.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
408 * Importing multiple modules via "import X, Y" is not allowed: use
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
409 separate import statements.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
410 * Importing multiple modules via "from X import ..." is allowed if using
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
411 parenthesis and one entry per line.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
412 * Only 1 relative import statement per import level ("from .", "from ..")
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
413 is allowed.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
414 * Relative imports from higher levels must occur before lower levels. e.g.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
415 "from .." must be before "from .".
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
416 * Imports from peer packages should use relative import (e.g. do not
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
417 "import mercurial.foo" from a "mercurial.*" module).
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
418 * Symbols can only be imported from specific modules (see
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
419 `allowsymbolimports`). For other modules, first import the module then
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
420 assign the symbol to a module-level variable. In addition, these imports
29208
cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules
Yuya Nishihara <yuya@tcha.org>
parents: 29207
diff changeset
421 must be performed before other local imports. This rule only
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
422 applies to import statements outside of any blocks.
34038
bc2535238de2 import-checker: allow relative import a module being checked
Jun Wu <quark@fb.com>
parents: 33920
diff changeset
423 * Relative imports from the standard library are not allowed, unless that
bc2535238de2 import-checker: allow relative import a module being checked
Jun Wu <quark@fb.com>
parents: 33920
diff changeset
424 library is also a local module.
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
425 * Certain modules must be aliased to alternate names to avoid aliasing
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
426 and readability problems. See `requirealias`.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
427 """
33890
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names
Augie Fackler <raf@durin42.com>
parents: 33877
diff changeset
428 if not isinstance(module, str):
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names
Augie Fackler <raf@durin42.com>
parents: 33877
diff changeset
429 module = module.decode('ascii')
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
430 topmodule = module.split('.')[0]
27272
69308357ecd1 import-checker: allow absolute imports of sub modules from local packages
Yuya Nishihara <yuya@tcha.org>
parents: 27018
diff changeset
431 fromlocal = fromlocalfunc(module, localmods)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
432
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
433 # Whether a local/non-stdlib import has been performed.
28330
f3fb24e36d61 import-checker: report local with stdlib late warning
timeless <timeless@mozdev.org>
parents: 27621
diff changeset
434 seenlocal = None
29208
cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules
Yuya Nishihara <yuya@tcha.org>
parents: 29207
diff changeset
435 # Whether a local/non-stdlib, non-symbol import has been seen.
cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules
Yuya Nishihara <yuya@tcha.org>
parents: 29207
diff changeset
436 seennonsymbollocal = False
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
437 # The last name to be imported (for sorting).
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
438 lastname = None
30590
74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29395
diff changeset
439 laststdlib = None
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
440 # Relative import levels encountered so far.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
441 seenlevels = set()
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
442
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
443 for node, newscope in walklocal(root):
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
444 def msg(fmt, *args):
26956
4b56214ebb7a import-checker: include lineno in warning message
Yuya Nishihara <yuya@tcha.org>
parents: 26955
diff changeset
445 return (fmt % args, node.lineno)
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
446 if newscope:
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
447 # Check for local imports in function
27272
69308357ecd1 import-checker: allow absolute imports of sub modules from local packages
Yuya Nishihara <yuya@tcha.org>
parents: 27018
diff changeset
448 for r in verify_modern_convention(module, node, localmods,
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
449 node.col_offset + 4):
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
450 yield r
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
451 elif isinstance(node, ast.Import):
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
452 # Disallow "import foo, bar" and require separate imports
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
453 # for each module.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
454 if len(node.names) > 1:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
455 yield msg('multiple imported names: %s',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
456 ', '.join(n.name for n in node.names))
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
457
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
458 name = node.names[0].name
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
459 asname = node.names[0].asname
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
460
30590
74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29395
diff changeset
461 stdlib = name in stdlib_modules
74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29395
diff changeset
462
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
463 # Ignore sorting rules on imports inside blocks.
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
464 if node.col_offset == root_col_offset:
30590
74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29395
diff changeset
465 if lastname and name < lastname and laststdlib == stdlib:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
466 yield msg('imports not lexically sorted: %s < %s',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
467 name, lastname)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
468
30590
74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29395
diff changeset
469 lastname = name
74eecb93c617 import-checker: do not enforce lexical sort accross stdlib/local boundary
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 29395
diff changeset
470 laststdlib = stdlib
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
471
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
472 # stdlib imports should be before local imports.
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
473 if stdlib and seenlocal and node.col_offset == root_col_offset:
28330
f3fb24e36d61 import-checker: report local with stdlib late warning
timeless <timeless@mozdev.org>
parents: 27621
diff changeset
474 yield msg('stdlib import "%s" follows local import: %s',
f3fb24e36d61 import-checker: report local with stdlib late warning
timeless <timeless@mozdev.org>
parents: 27621
diff changeset
475 name, seenlocal)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
476
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
477 if not stdlib:
28330
f3fb24e36d61 import-checker: report local with stdlib late warning
timeless <timeless@mozdev.org>
parents: 27621
diff changeset
478 seenlocal = name
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
479
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
480 # Import of sibling modules should use relative imports.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
481 topname = name.split('.')[0]
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
482 if topname == topmodule:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
483 yield msg('import should be relative: %s', name)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
484
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
485 if name in requirealias and asname != requirealias[name]:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
486 yield msg('%s module must be "as" aliased to %s',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
487 name, requirealias[name])
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
488
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
489 elif isinstance(node, ast.ImportFrom):
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
490 # Resolve the full imported module name.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
491 if node.level > 0:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
492 fullname = '.'.join(module.split('.')[:-node.level])
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
493 if node.module:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
494 fullname += '.%s' % node.module
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
495 else:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
496 assert node.module
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
497 fullname = node.module
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
498
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
499 topname = fullname.split('.')[0]
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
500 if topname == topmodule:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
501 yield msg('import should be relative: %s', fullname)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
502
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
503 # __future__ is special since it needs to come first and use
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
504 # symbol import.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
505 if fullname != '__future__':
34038
bc2535238de2 import-checker: allow relative import a module being checked
Jun Wu <quark@fb.com>
parents: 33920
diff changeset
506 if not fullname or (
bc2535238de2 import-checker: allow relative import a module being checked
Jun Wu <quark@fb.com>
parents: 33920
diff changeset
507 fullname in stdlib_modules
bc2535238de2 import-checker: allow relative import a module being checked
Jun Wu <quark@fb.com>
parents: 33920
diff changeset
508 and fullname not in localmods
bc2535238de2 import-checker: allow relative import a module being checked
Jun Wu <quark@fb.com>
parents: 33920
diff changeset
509 and fullname + '.__init__' not in localmods):
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
510 yield msg('relative import of stdlib module')
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
511 else:
28330
f3fb24e36d61 import-checker: report local with stdlib late warning
timeless <timeless@mozdev.org>
parents: 27621
diff changeset
512 seenlocal = fullname
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
513
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
514 # Direct symbol import is only allowed from certain modules and
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
515 # must occur before non-symbol imports.
29207
a09098c61fea import-checker: always build a list of imported symbols
Yuya Nishihara <yuya@tcha.org>
parents: 29122
diff changeset
516 found = fromlocal(node.module, node.level)
a09098c61fea import-checker: always build a list of imported symbols
Yuya Nishihara <yuya@tcha.org>
parents: 29122
diff changeset
517 if found and found[2]: # node.module is a package
a09098c61fea import-checker: always build a list of imported symbols
Yuya Nishihara <yuya@tcha.org>
parents: 29122
diff changeset
518 prefix = found[0] + '.'
32419
d02888308235 import-checker: add a way to directly import certain symbols
Siddharth Agarwal <sid0@fb.com>
parents: 32374
diff changeset
519 symbols = (n.name for n in node.names
d02888308235 import-checker: add a way to directly import certain symbols
Siddharth Agarwal <sid0@fb.com>
parents: 32374
diff changeset
520 if not fromlocal(prefix + n.name))
29207
a09098c61fea import-checker: always build a list of imported symbols
Yuya Nishihara <yuya@tcha.org>
parents: 29122
diff changeset
521 else:
32419
d02888308235 import-checker: add a way to directly import certain symbols
Siddharth Agarwal <sid0@fb.com>
parents: 32374
diff changeset
522 symbols = (n.name for n in node.names)
d02888308235 import-checker: add a way to directly import certain symbols
Siddharth Agarwal <sid0@fb.com>
parents: 32374
diff changeset
523 symbols = [sym for sym in symbols if sym not in directsymbols]
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
524 if node.module and node.col_offset == root_col_offset:
27272
69308357ecd1 import-checker: allow absolute imports of sub modules from local packages
Yuya Nishihara <yuya@tcha.org>
parents: 27018
diff changeset
525 if symbols and fullname not in allowsymbolimports:
27273
5d5b98346fc2 import-checker: tell which symbol causes "direct symbol import"
Yuya Nishihara <yuya@tcha.org>
parents: 27272
diff changeset
526 yield msg('direct symbol import %s from %s',
5d5b98346fc2 import-checker: tell which symbol causes "direct symbol import"
Yuya Nishihara <yuya@tcha.org>
parents: 27272
diff changeset
527 ', '.join(symbols), fullname)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
528
29208
cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules
Yuya Nishihara <yuya@tcha.org>
parents: 29207
diff changeset
529 if symbols and seennonsymbollocal:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
530 yield msg('symbol import follows non-symbol import: %s',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
531 fullname)
29208
cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules
Yuya Nishihara <yuya@tcha.org>
parents: 29207
diff changeset
532 if not symbols and fullname not in stdlib_modules:
cba8bc11ed10 import-checker: extend check of symbol-import order to all local modules
Yuya Nishihara <yuya@tcha.org>
parents: 29207
diff changeset
533 seennonsymbollocal = True
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
534
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
535 if not node.module:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
536 assert node.level
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
537
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
538 # Only allow 1 group per level.
26965
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
539 if (node.level in seenlevels
1fa66d3ad28d import-checker: reset context to verify convention in function scope
Yuya Nishihara <yuya@tcha.org>
parents: 26964
diff changeset
540 and node.col_offset == root_col_offset):
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
541 yield msg('multiple "from %s import" statements',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
542 '.' * node.level)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
543
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
544 # Higher-level groups come before lower-level groups.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
545 if any(node.level > l for l in seenlevels):
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
546 yield msg('higher-level import should come first: %s',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
547 fullname)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
548
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
549 seenlevels.add(node.level)
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
550
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
551 # Entries in "from .X import ( ... )" lists must be lexically
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
552 # sorted.
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
553 lastentryname = None
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
554
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
555 for n in node.names:
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
556 if lastentryname and n.name < lastentryname:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
557 yield msg('imports from %s not lexically sorted: %s < %s',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
558 fullname, n.name, lastentryname)
25703
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
559
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
560 lastentryname = n.name
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
561
1a6a117d0b95 import-checker: establish modern import convention
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25702
diff changeset
562 if n.name in requirealias and n.asname != requirealias[n.name]:
26955
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
563 yield msg('%s from %s must be "as" aliased to %s',
c4114e335b49 import-checker: extract function to generate a formatted warning
Yuya Nishihara <yuya@tcha.org>
parents: 26954
diff changeset
564 n.name, fullname, requirealias[n.name])
25702
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
565
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
566 def verify_stdlib_on_own_line(root):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
567 """Given some python source, verify that stdlib imports are done
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
568 in separate statements from relative local module imports.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
569
25702
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
570 >>> list(verify_stdlib_on_own_line(ast.parse('import sys, foo')))
26956
4b56214ebb7a import-checker: include lineno in warning message
Yuya Nishihara <yuya@tcha.org>
parents: 26955
diff changeset
571 [('mixed imports\\n stdlib: sys\\n relative: foo', 1)]
25702
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
572 >>> list(verify_stdlib_on_own_line(ast.parse('import sys, os')))
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
573 []
25702
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
574 >>> list(verify_stdlib_on_own_line(ast.parse('import foo, bar')))
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
575 []
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
576 """
25702
ab2c5163900e import-checker: establish new function for verifying import conventions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25701
diff changeset
577 for node in ast.walk(root):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
578 if isinstance(node, ast.Import):
20386
a05d31b040d7 import-checker: show stdlib and relative imports separately
Mads Kiilerich <madski@unity3d.com>
parents: 20383
diff changeset
579 from_stdlib = {False: [], True: []}
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
580 for n in node.names:
20386
a05d31b040d7 import-checker: show stdlib and relative imports separately
Mads Kiilerich <madski@unity3d.com>
parents: 20383
diff changeset
581 from_stdlib[n.name in stdlib_modules].append(n.name)
a05d31b040d7 import-checker: show stdlib and relative imports separately
Mads Kiilerich <madski@unity3d.com>
parents: 20383
diff changeset
582 if from_stdlib[True] and from_stdlib[False]:
a05d31b040d7 import-checker: show stdlib and relative imports separately
Mads Kiilerich <madski@unity3d.com>
parents: 20383
diff changeset
583 yield ('mixed imports\n stdlib: %s\n relative: %s' %
a05d31b040d7 import-checker: show stdlib and relative imports separately
Mads Kiilerich <madski@unity3d.com>
parents: 20383
diff changeset
584 (', '.join(sorted(from_stdlib[True])),
26956
4b56214ebb7a import-checker: include lineno in warning message
Yuya Nishihara <yuya@tcha.org>
parents: 26955
diff changeset
585 ', '.join(sorted(from_stdlib[False]))), node.lineno)
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
586
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
587 class CircularImport(Exception):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
588 pass
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
589
24490
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
590 def checkmod(mod, imports):
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
591 shortest = {}
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
592 visit = [[mod]]
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
593 while visit:
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
594 path = visit.pop(0)
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
595 for i in sorted(imports.get(path[-1], [])):
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
596 if len(path) < shortest.get(i, 1000):
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
597 shortest[i] = len(path)
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
598 if i in path:
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
599 if i == path[0]:
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
600 raise CircularImport(path)
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
601 continue
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
602 visit.append(path + [i])
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
603
20038
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
604 def rotatecycle(cycle):
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
605 """arrange a cycle so that the lexicographically first module listed first
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
606
24488
4b3fc46097f7 import-checker: drop duplicate element from cycle
Matt Mackall <mpm@selenic.com>
parents: 24487
diff changeset
607 >>> rotatecycle(['foo', 'bar'])
20038
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
608 ['bar', 'foo', 'bar']
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
609 """
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
610 lowest = min(cycle)
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
611 idx = cycle.index(lowest)
24488
4b3fc46097f7 import-checker: drop duplicate element from cycle
Matt Mackall <mpm@selenic.com>
parents: 24487
diff changeset
612 return cycle[idx:] + cycle[:idx] + [lowest]
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
613
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
614 def find_cycles(imports):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
615 """Find cycles in an already-loaded import graph.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
616
25175
10e6c4b7121b import-checker: don't treat modules as relative one if not found
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25174
diff changeset
617 All module names recorded in `imports` should be absolute one.
10e6c4b7121b import-checker: don't treat modules as relative one if not found
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25174
diff changeset
618
28703
a274c4f9087a py3: use print_function in import-checker
timeless <timeless@mozdev.org>
parents: 28702
diff changeset
619 >>> from __future__ import print_function
25175
10e6c4b7121b import-checker: don't treat modules as relative one if not found
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25174
diff changeset
620 >>> imports = {'top.foo': ['top.bar', 'os.path', 'top.qux'],
10e6c4b7121b import-checker: don't treat modules as relative one if not found
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25174
diff changeset
621 ... 'top.bar': ['top.baz', 'sys'],
10e6c4b7121b import-checker: don't treat modules as relative one if not found
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25174
diff changeset
622 ... 'top.baz': ['top.foo'],
10e6c4b7121b import-checker: don't treat modules as relative one if not found
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25174
diff changeset
623 ... 'top.qux': ['top.foo']}
28703
a274c4f9087a py3: use print_function in import-checker
timeless <timeless@mozdev.org>
parents: 28702
diff changeset
624 >>> print('\\n'.join(sorted(find_cycles(imports))))
24487
642d245ff537 import-checker: fix rotatecycle
Matt Mackall <mpm@selenic.com>
parents: 22975
diff changeset
625 top.bar -> top.baz -> top.foo -> top.bar
642d245ff537 import-checker: fix rotatecycle
Matt Mackall <mpm@selenic.com>
parents: 22975
diff changeset
626 top.foo -> top.qux -> top.foo
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
627 """
24491
784b278b349c import-checker: rotatecycle is actually the canonical cycle key
Matt Mackall <mpm@selenic.com>
parents: 24490
diff changeset
628 cycles = set()
28704
1fa6fdb72275 py3: handle iter/iterkeys+iteritems python3 divergence in import-checker
timeless <timeless@mozdev.org>
parents: 28703
diff changeset
629 for mod in sorted(imports.keys()):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
630 try:
24490
fb4639d5268e import-checker: make search algorithm non-recursive breadth-first
Matt Mackall <mpm@selenic.com>
parents: 24489
diff changeset
631 checkmod(mod, imports)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25175
diff changeset
632 except CircularImport as e:
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
633 cycle = e.args[0]
24491
784b278b349c import-checker: rotatecycle is actually the canonical cycle key
Matt Mackall <mpm@selenic.com>
parents: 24490
diff changeset
634 cycles.add(" -> ".join(rotatecycle(cycle)))
784b278b349c import-checker: rotatecycle is actually the canonical cycle key
Matt Mackall <mpm@selenic.com>
parents: 24490
diff changeset
635 return cycles
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
636
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
637 def _cycle_sortkey(c):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
638 return len(c), c
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
639
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
640 def embedded(f, modname, src):
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
641 """Extract embedded python code
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
642
33877
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
643 >>> def _forcestr(thing):
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
644 ... if not isinstance(thing, str):
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
645 ... return thing.decode('ascii')
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
646 ... return thing
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
647 >>> def test(fn, lines):
33877
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
648 ... for s, m, f, l in embedded(fn, b"example", lines):
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
649 ... print("%s %s %d" % (_forcestr(m), _forcestr(f), l))
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
650 ... print(repr(_forcestr(s)))
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
651 >>> lines = [
33877
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
652 ... b'comment',
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
653 ... b' >>> from __future__ import print_function',
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
654 ... b" >>> ' multiline",
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
655 ... b" ... string'",
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
656 ... b' ',
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
657 ... b'comment',
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
658 ... b' $ cat > foo.py <<EOF',
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
659 ... b' > from __future__ import print_function',
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
660 ... b' > EOF',
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
661 ... ]
33877
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
662 >>> test(b"example.t", lines)
40095
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 39329
diff changeset
663 example[2] doctest.py 1
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 39329
diff changeset
664 "from __future__ import print_function\\n' multiline\\nstring'\\n\\n"
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 39329
diff changeset
665 example[8] foo.py 7
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
666 'from __future__ import print_function\\n'
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
667 """
40095
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 39329
diff changeset
668 errors = []
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 39329
diff changeset
669 for name, starts, ends, code in testparseutil.pyembedded(f, src, errors):
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 39329
diff changeset
670 if not name:
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 39329
diff changeset
671 # use 'doctest.py', in order to make already existing
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 39329
diff changeset
672 # doctest above pass instantly
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 39329
diff changeset
673 name = 'doctest.py'
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 39329
diff changeset
674 # "starts" is "line number" (1-origin), but embedded() is
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 39329
diff changeset
675 # expected to return "line offset" (0-origin). Therefore, this
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 39329
diff changeset
676 # yields "starts - 1".
40480
99d5424eedc8 contrib: fix import-checker to not b'' module names on Python 3
Augie Fackler <augie@google.com>
parents: 40095
diff changeset
677 if not isinstance(modname, str):
99d5424eedc8 contrib: fix import-checker to not b'' module names on Python 3
Augie Fackler <augie@google.com>
parents: 40095
diff changeset
678 modname = modname.decode('utf8')
40095
7288838bec1f import-checker: use testparseutil.embedded() to centralize detection logic
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 39329
diff changeset
679 yield code, "%s[%d]" % (modname, starts), name, starts - 1
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
680
28919
a94f34306bb9 import-checker: refactor source reading
timeless <timeless@mozdev.org>
parents: 28713
diff changeset
681 def sources(f, modname):
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
682 """Yields possibly multiple sources from a filepath
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
683
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
684 input: filepath, modulename
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
685 yields: script(string), modulename, filepath, linenumber
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
686
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
687 For embedded scripts, the modulename and filepath will be different
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
688 from the function arguments. linenumber is an offset relative to
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
689 the input file.
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
690 """
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
691 py = False
29234
393aef802535 tests: enable import checker for all python files (including no .py files)
Yuya Nishihara <yuya@tcha.org>
parents: 29211
diff changeset
692 if not f.endswith('.t'):
33877
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
693 with open(f, 'rb') as src:
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
694 yield src.read(), modname, f, 0
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
695 py = True
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
696 if py or f.endswith('.t'):
33877
bcf53149ebce contrib: make import checker always think in terms of bytes
Augie Fackler <raf@durin42.com>
parents: 33876
diff changeset
697 with open(f, 'rb') as src:
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
698 for script, modname, t, line in embedded(f, modname, src):
40481
90517fad4293 contrib: tweak import-checker to always use bytes for module names
Augie Fackler <augie@google.com>
parents: 40480
diff changeset
699 yield script, modname.encode('utf8'), t, line
28919
a94f34306bb9 import-checker: refactor source reading
timeless <timeless@mozdev.org>
parents: 28713
diff changeset
700
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
701 def main(argv):
25063
723e364488f4 import-checker: add xargs like mode
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24669
diff changeset
702 if len(argv) < 2 or (argv[1] == '-' and len(argv) > 2):
28703
a274c4f9087a py3: use print_function in import-checker
timeless <timeless@mozdev.org>
parents: 28702
diff changeset
703 print('Usage: %s {-|file [file] [file] ...}')
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
704 return 1
25063
723e364488f4 import-checker: add xargs like mode
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24669
diff changeset
705 if argv[1] == '-':
723e364488f4 import-checker: add xargs like mode
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24669
diff changeset
706 argv = argv[:1]
723e364488f4 import-checker: add xargs like mode
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24669
diff changeset
707 argv.extend(l.rstrip() for l in sys.stdin.readlines())
32508
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32507
diff changeset
708 localmodpaths = {}
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
709 used_imports = {}
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
710 any_errors = False
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
711 for source_path in argv[1:]:
32374
194b0f781132 import-checker: drop workaround for pure modules
Yuya Nishihara <yuya@tcha.org>
parents: 32372
diff changeset
712 modname = dotted_name_of_path(source_path)
32508
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32507
diff changeset
713 localmodpaths[modname] = source_path
32509
a025ec43856c import-checker: guess names of C extension modules
Yuya Nishihara <yuya@tcha.org>
parents: 32508
diff changeset
714 localmods = populateextmods(localmodpaths)
32508
4c712b90c60a import-checker: convert localmods to a set of module names
Yuya Nishihara <yuya@tcha.org>
parents: 32507
diff changeset
715 for localmodname, source_path in sorted(localmodpaths.items()):
33890
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names
Augie Fackler <raf@durin42.com>
parents: 33877
diff changeset
716 if not isinstance(localmodname, bytes):
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names
Augie Fackler <raf@durin42.com>
parents: 33877
diff changeset
717 # This is only safe because all hg's files are ascii
3de9a2df6411 contrib: have import-checker work mostly with native strings for mod names
Augie Fackler <raf@durin42.com>
parents: 33877
diff changeset
718 localmodname = localmodname.encode('ascii')
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
719 for src, modname, name, line in sources(source_path, localmodname):
28920
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
720 try:
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
721 used_imports[modname] = sorted(
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
722 imported_modules(src, modname, name, localmods,
28920
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
723 ignore_nested=True))
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
724 for error, lineno in verify_import_convention(modname, src,
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
725 localmods):
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
726 any_errors = True
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
727 print('%s:%d: %s' % (source_path, lineno + line, error))
28920
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
728 except SyntaxError as e:
cdf331b54eb8 import-checker: track SyntaxErrors
timeless <timeless@mozdev.org>
parents: 28919
diff changeset
729 print('%s:%d: SyntaxError: %s' %
28922
4ec62a084e5c import-checker: parse python code from .t files
timeless <timeless@mozdev.org>
parents: 28921
diff changeset
730 (source_path, e.lineno + line, e))
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
731 cycles = find_cycles(used_imports)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
732 if cycles:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
733 firstmods = set()
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
734 for c in sorted(cycles, key=_cycle_sortkey):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
735 first = c.split()[0]
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
736 # As a rough cut, ignore any cycle that starts with the
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
737 # same module as some other cycle. Otherwise we see lots
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
738 # of cycles that are effectively duplicates.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
739 if first in firstmods:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
740 continue
28703
a274c4f9087a py3: use print_function in import-checker
timeless <timeless@mozdev.org>
parents: 28702
diff changeset
741 print('Import cycle:', c)
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
742 firstmods.add(first)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
743 any_errors = True
25731
cd1daab5d036 import-checker.py: exit with code 0 if no error is detected
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 25703
diff changeset
744 return any_errors != 0
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
745
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
746 if __name__ == '__main__':
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
747 sys.exit(int(main(sys.argv)))