annotate hgdemandimport/demandimportpy3.py @ 42278:8dc22a209420

automation: wait for instance profiles and roles Otherwise there is a race condition between creating the resources and us attempting to use them / them becoming available. The role waiter API was recently introduced, so we had to upgrade the boto3 package to get it. Other packages were also updated to latest versions just because. Even with this change, I still run into issues with the IAM instance profile not being available when we attempt to create an EC2 instance using a just-created profile. I'm not sure what's going on. Possibly a bug on Amazon's end. But the new behavior is "more correct." Differential Revision: https://phab.mercurial-scm.org/D6286
author Gregory Szorc <gregory.szorc@gmail.com>
date Sat, 27 Apr 2019 11:38:58 -0700
parents 670eb4fa1b86
children adb636392b3f
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
1 # demandimportpy3 - global demand-loading of modules for Mercurial
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
2 #
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
3 # Copyright 2017 Facebook Inc.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
4 #
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
7
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
8 """Lazy loading for Python 3.6 and above.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
9
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
10 This uses the new importlib finder/loader functionality available in Python 3.5
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
11 and up. The code reuses most of the mechanics implemented inside importlib.util,
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
12 but with a few additions:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
13
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
14 * Allow excluding certain modules from lazy imports.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
15 * Expose an interface that's substantially the same as demandimport for
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
16 Python 2.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
17
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
18 This also has some limitations compared to the Python 2 implementation:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
19
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
20 * Much of the logic is per-package, not per-module, so any packages loaded
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
21 before demandimport is enabled will not be lazily imported in the future. In
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
22 practice, we only expect builtins to be loaded before demandimport is
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
23 enabled.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
24 """
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
25
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
26 # This line is unnecessary, but it satisfies test-check-py3-compat.t.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
27 from __future__ import absolute_import
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
28
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
29 import contextlib
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
30 import importlib.abc
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
31 import importlib.machinery
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
32 import importlib.util
33898
3595e4e0ae57 demandimportpy3: update to pass import checker
Augie Fackler <raf@durin42.com>
parents: 33859
diff changeset
33 import sys
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
34
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
35 _deactivated = False
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
36
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
37 class _lazyloaderex(importlib.util.LazyLoader):
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
38 """This is a LazyLoader except it also follows the _deactivated global and
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
39 the ignore list.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
40 """
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
41 def exec_module(self, module):
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
42 """Make the module load lazily."""
37843
670eb4fa1b86 demandimport: make module ignores a set (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35524
diff changeset
43 if _deactivated or module.__name__ in ignores:
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
44 self.loader.exec_module(module)
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
45 else:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
46 super().exec_module(module)
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
47
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
48 # This is 3.6+ because with Python 3.5 it isn't possible to lazily load
35524
fcb1ecf2bef7 hgdemandimport: use correct hyperlink to python-bug in comments (issue5765)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33898
diff changeset
49 # extensions. See the discussion in https://bugs.python.org/issue26186 for more.
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
50 _extensions_loader = _lazyloaderex.factory(
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
51 importlib.machinery.ExtensionFileLoader)
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
52 _bytecode_loader = _lazyloaderex.factory(
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
53 importlib.machinery.SourcelessFileLoader)
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
54 _source_loader = _lazyloaderex.factory(importlib.machinery.SourceFileLoader)
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
55
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
56 def _makefinder(path):
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
57 return importlib.machinery.FileFinder(
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
58 path,
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
59 # This is the order in which loaders are passed in in core Python.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
60 (_extensions_loader, importlib.machinery.EXTENSION_SUFFIXES),
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
61 (_source_loader, importlib.machinery.SOURCE_SUFFIXES),
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
62 (_bytecode_loader, importlib.machinery.BYTECODE_SUFFIXES),
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
63 )
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
64
37843
670eb4fa1b86 demandimport: make module ignores a set (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35524
diff changeset
65 ignores = set()
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
66
37843
670eb4fa1b86 demandimport: make module ignores a set (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35524
diff changeset
67 def init(ignoreset):
670eb4fa1b86 demandimport: make module ignores a set (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35524
diff changeset
68 global ignores
670eb4fa1b86 demandimport: make module ignores a set (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35524
diff changeset
69 ignores = ignoreset
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
70
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
71 def isenabled():
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
72 return _makefinder in sys.path_hooks and not _deactivated
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
73
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
74 def disable():
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
75 try:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
76 while True:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
77 sys.path_hooks.remove(_makefinder)
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
78 except ValueError:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
79 pass
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
80
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
81 def enable():
33859
8fb5212652ec demandimport: move HGDEMANDIMPORT test to __init__.py
Jun Wu <quark@fb.com>
parents: 32423
diff changeset
82 sys.path_hooks.insert(0, _makefinder)
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
83
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
84 @contextlib.contextmanager
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
85 def deactivated():
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
86 # This implementation is a bit different from Python 2's. Python 3
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
87 # maintains a per-package finder cache in sys.path_importer_cache (see
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
88 # PEP 302). This means that we can't just call disable + enable.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
89 # If we do that, in situations like:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
90 #
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
91 # demandimport.enable()
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
92 # ...
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
93 # from foo.bar import mod1
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
94 # with demandimport.deactivated():
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
95 # from foo.bar import mod2
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
96 #
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
97 # mod2 will be imported lazily. (The converse also holds -- whatever finder
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
98 # first gets cached will be used.)
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
99 #
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
100 # Instead, have a global flag the LazyLoader can use.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
101 global _deactivated
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
102 demandenabled = isenabled()
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
103 if demandenabled:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
104 _deactivated = True
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
105 try:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
106 yield
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
107 finally:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
108 if demandenabled:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
109 _deactivated = False