changeset 43678:52e4bfebc4ba

setup: conditionalize access to `sys.dllhandle` when building extensions

This code is only run on Windows, and was crashing PyOxidizer when running in
`setup-py-install` mode.  Now an oxidized binary can be built by simply pointing
to setup.py.

Something is slightly different now that it's not being built from a virtualenv.
Previously, `hg version` could print to the screen, but now it aborts saying
"Incorrect function".  But I can see the output if redirected to a file, and
it's not complaining about missing C extensions, so I think those are loading
now (unlike from the virtualenv).  The interesting this about this incorrect
function output is that it failed when initially built.  I then went back and
did a `make clean` and `make local` with py3 and then py2 to ensure I didn't
break the existing code.  At that point I ran the oxidized executable again and
it was able to print to the screen normally!  So I ran `pyoxidizer build` again,
it only output the following, and then running the executable failed to output
again:

    (pyO2_venv) C:\Users\Matt\hg3\hg_pyO2>pyoxidizer build
        Finished dev [unoptimized + debuginfo] target(s) in 0.12s
    packaging application into C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug
    purging C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug
    copying C:/Users/Matt/hg3/hg_pyO2\build\target\x86_64-pc-windows-msvc\debug\hg_pyO2.exe to
        C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug\hg_pyO2.exe
    resolving packaging state...
    writing license for [...]
    hg_pyO2 packaged into C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug
    executable path: C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug\hg_pyO2.exe

Differential Revision: https://phab.mercurial-scm.org/D7444

--- a/setup.py	Sat Nov 16 11:48:47 2019 -0500
+++ b/setup.py	Sat Nov 16 12:19:43 2019 -0500
@@ -713,30 +713,34 @@
             self.compiler.compiler_so = self.compiler.compiler  # no -mdll
             self.compiler.dll_libraries = []  # no -lmsrvc90
 
-        # Different Python installs can have different Python library
-        # names. e.g. the official CPython distribution uses pythonXY.dll
-        # and MinGW uses libpythonX.Y.dll.
-        _kernel32 = ctypes.windll.kernel32
-        _kernel32.GetModuleFileNameA.argtypes = [
-            ctypes.c_void_p,
-            ctypes.c_void_p,
-            ctypes.c_ulong,
-        ]
-        _kernel32.GetModuleFileNameA.restype = ctypes.c_ulong
-        size = 1000
-        buf = ctypes.create_string_buffer(size + 1)
-        filelen = _kernel32.GetModuleFileNameA(
-            sys.dllhandle, ctypes.byref(buf), size
-        )
+        pythonlib = None
 
-        if filelen > 0 and filelen != size:
-            dllbasename = os.path.basename(buf.value)
-            if not dllbasename.lower().endswith(b'.dll'):
-                raise SystemExit(
-                    'Python DLL does not end with .dll: %s' % dllbasename
-                )
-            pythonlib = dllbasename[:-4]
-        else:
+        if getattr(sys, 'dllhandle', None):
+            # Different Python installs can have different Python library
+            # names. e.g. the official CPython distribution uses pythonXY.dll
+            # and MinGW uses libpythonX.Y.dll.
+            _kernel32 = ctypes.windll.kernel32
+            _kernel32.GetModuleFileNameA.argtypes = [
+                ctypes.c_void_p,
+                ctypes.c_void_p,
+                ctypes.c_ulong,
+            ]
+            _kernel32.GetModuleFileNameA.restype = ctypes.c_ulong
+            size = 1000
+            buf = ctypes.create_string_buffer(size + 1)
+            filelen = _kernel32.GetModuleFileNameA(
+                sys.dllhandle, ctypes.byref(buf), size
+            )
+
+            if filelen > 0 and filelen != size:
+                dllbasename = os.path.basename(buf.value)
+                if not dllbasename.lower().endswith(b'.dll'):
+                    raise SystemExit(
+                        'Python DLL does not end with .dll: %s' % dllbasename
+                    )
+                pythonlib = dllbasename[:-4]
+
+        if not pythonlib:
             log.warn(
                 'could not determine Python DLL filename; assuming pythonXY'
             )