annotate contrib/packaging/hgpackaging/wix.py @ 41921:4371f543efda

wix: functionality to automate building WiX installers

Like we did for Inno Setup, we want to make it easier to
produce WiX installers. This commit does that.

We introduce a new hgpackaging.wix module for performing
all the high-level tasks required to produce WiX installers.
This required miscellaneous enhancements to existing code in
hgpackaging, including support for signing binaries.

A new build.py script for calling into the module APIs has been
created. It behaves very similarly to the Inno Setup build.py
script.

Unlike Inno Setup, we didn't have code in the repo previously
to generate WiX installers. It appears that all existing
automation for building WiX installers lives in the
https://bitbucket.org/tortoisehg/thg-winbuild repository - most
notably in its setup.py file. My strategy for inventing the
code in this commit was to step through the code in that repo's
setup.py and observe what it was doing. Despite the length of
setup.py in that repository, the actual amount of steps required
to produce a WiX installer is actually quite low. It consists
of a basic py2exe build plus invocations of candle.exe and
light.exe to produce the MSI.

One rabbit hole that gave me fits was locating the Visual Studio
9 C Runtime merge modules. These merge modules are only present
on your system if you have a full Visual Studio 2008 installation.
Fortunately, I have a copy of Visual Studio 2008 and was able
to install all the required updates. I then uploaded these merge
modules to a personal repository on GitHub. That is where the
added code references them from. We probably don't need to
ship the merge modules. But that is for another day.

The installs from the MSIs produced with the new automation
differ from the last official MSI in the following ways:

* Our HTML manual pages have UNIX line endings instead of Windows.
* We ship modules in the mercurial.pure package. It appears the
  upstream packaging code is not including this package due to
  omission (they supply an explicit list of packages that has
  drifted out of sync with our setup.py).
* We do not ship various distutils.* modules. This is because
  virtualenvs have a custom distutils/__init__.py that automagically
  imports distutils from its original location and py2exe gets
  confused by this. We don't use distutils in core Mercurial and
  don't provide a usable python.exe, so this omission should be
  acceptable.
* The version of the enum package is different and we ship
  an enum.pyc instead of an enum/__init__.py.
* The version of the docutils package is different and we
  ship a different set of files.
* The version of Sphinx is drastically newer and we ship a
  number of files the old version did not. (I'm not sure why
  we ship Sphinx - I think it is a side-effect of the way the
  THG code was installing dependencies.)
* We ship the idna package (dependent of requests which is a
  dependency of newer versions of Sphinx).
* The version of imagesize is different and we ship an
  imagesize.pyc instead of an imagesize/__init__.pyc.
* The version of the jinja2 package is different and the sets
  of files differs.
* We ship the packaging package, which is a dependency for Sphinx.
* The version of the pygments package is different and the sets
  of files differs.
* We ship the requests package, which is a dependency for Sphinx.
* We ship the snowballstemmer package, which is a dependency for
  Sphinx.
* We ship the urllib3 package, which is a dependency for requests,
  which is a dependency for Sphinx.
* We ship a newer version of the futures package, which includes a
  handful of extra modules that match Python 3 module names.

# no-check-commit because foo_bar naming

Differential Revision: https://phab.mercurial-scm.org/D6097

# wix.py - WiX installer functionality
#
# Copyright 2019 Gregory Szorc <[e-mail]>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.

# no-check-code because Python 3 native.

import os
import pathlib
import re
import subprocess

from .downloads import (
    download_entry,
)
from .py2exe import (
    build_py2exe,
)
from .util import (
    extract_zip_to_directory,
    sign_with_signtool,
)


SUPPORT_WXS = [
    ('contrib.wxs', r'contrib'),
    ('dist.wxs', r'dist'),
    ('doc.wxs', r'doc'),
    ('help.wxs', r'mercurial\help'),
    ('i18n.wxs', r'i18n'),
    ('locale.wxs', r'mercurial\locale'),
    ('templates.wxs', r'mercurial\templates'),
]


EXTRA_PACKAGES = {
    'distutils',
    'enum',
    'imagesize',
    'pygments',
    'sphinx',
}


EXCLUDES = {
    # Python 3 only.
    'jinja2.asyncsupport',
}


def find_version(source_dir: pathlib.Path):
    version_py = source_dir / 'mercurial' / '__version__.py'

    with version_py.open('r', encoding='utf-8') as fh:
        source = fh.read().strip()

    m = re.search('version = b"(.*)"', source)
    return m.group(1)


def normalize_version(version):
    """Normalize Mercurial version string so WiX accepts it.

    Version strings have to be numeric X.Y.Z.
    """

    if '+' in version:
        version, extra = version.split('+', 1)
    else:
        extra = None

    # 4.9rc0
    if version[:-1].endswith('rc'):
        version = version[:-3]

    versions = [int(v) for v in version.split('.')]
    while len(versions) < 3:
        versions.append(0)

    major, minor, build = versions[:3]

    if extra:
        # <commit count>-<hash>+<date>
        build = int(extra.split('-')[0])

    return '.'.join('%d' % x for x in (major, minor, build))


def ensure_vc90_merge_modules(build_dir):
    x86 = (
        download_entry('vc9-crt-x86-msm', build_dir,
                       local_name='microsoft.vcxx.crt.x86_msm.msm')[0],
        download_entry('vc9-crt-x86-msm-policy', build_dir,
                       local_name='policy.x.xx.microsoft.vcxx.crt.x86_msm.msm')[0]
    )

    x64 = (
        download_entry('vc9-crt-x64-msm', build_dir,
                       local_name='microsoft.vcxx.crt.x64_msm.msm')[0],
        download_entry('vc9-crt-x64-msm-policy', build_dir,
                       local_name='policy.x.xx.microsoft.vcxx.crt.x64_msm.msm')[0]
    )
    return {
        'x86': x86,
        'x64': x64,
    }


def run_candle(wix, cwd, wxs, source_dir, defines=None):
    args = [
        str(wix / 'candle.exe'),
        '-nologo',
        str(wxs),
        '-dSourceDir=%s' % source_dir,
    ]

    if defines:
        args.extend('-d%s=%s' % define for define in sorted(defines.items()))

    subprocess.run(args, cwd=str(cwd), check=True)


def make_post_build_signing_fn(name, subject_name=None, cert_path=None,
                               cert_password=None, timestamp_url=None):
    """Create a callable that will use signtool to sign hg.exe."""

    def post_build_sign(source_dir, build_dir, dist_dir, version):
        description = '%s %s' % (name, version)

        sign_with_signtool(dist_dir / 'hg.exe', description,
                           subject_name=subject_name, cert_path=cert_path,
                           cert_password=cert_password,
                           timestamp_url=timestamp_url)

    return post_build_sign


def build_installer(source_dir: pathlib.Path, python_exe: pathlib.Path,
                    msi_name='mercurial', version=None, post_build_fn=None):
    """Build a WiX MSI installer.

    ``source_dir`` is the path to the Mercurial source tree to use.
    ``arch`` is the target architecture. either ``x86`` or ``x64``.
    ``python_exe`` is the path to the Python executable to use/bundle.
    ``version`` is the Mercurial version string. If not defined,
    ``mercurial/__version__.py`` will be consulted.
    ``post_build_fn`` is a callable that will be called after building
    Mercurial but before invoking WiX. It can be used to e.g. facilitate
    signing. It is passed the paths to the Mercurial source, build, and
    dist directories and the resolved Mercurial version.
    """
    arch = 'x64' if r'\x64' in os.environ.get('LIB', '') else 'x86'

    hg_build_dir = source_dir / 'build'
    dist_dir = source_dir / 'dist'

    requirements_txt = (source_dir / 'contrib' / 'packaging' /
                        'wix' / 'requirements.txt')

    build_py2exe(source_dir, hg_build_dir,
                 python_exe, 'wix', requirements_txt,
                 extra_packages=EXTRA_PACKAGES, extra_excludes=EXCLUDES)

    version = version or normalize_version(find_version(source_dir))
    print('using version string: %s' % version)

    if post_build_fn:
        post_build_fn(source_dir, hg_build_dir, dist_dir, version)

    build_dir = hg_build_dir / ('wix-%s' % arch)

    build_dir.mkdir(exist_ok=True)

    wix_pkg, wix_entry = download_entry('wix', hg_build_dir)
    wix_path = hg_build_dir / ('wix-%s' % wix_entry['version'])

    if not wix_path.exists():
        extract_zip_to_directory(wix_pkg, wix_path)

    ensure_vc90_merge_modules(hg_build_dir)

    source_build_rel = pathlib.Path(os.path.relpath(source_dir, build_dir))

    defines = {'Platform': arch}

    for wxs, rel_path in SUPPORT_WXS:
        wxs = source_dir / 'contrib' / 'packaging' / 'wix' / wxs
        wxs_source_dir = source_dir / rel_path
        run_candle(wix_path, build_dir, wxs, wxs_source_dir, defines=defines)

    source = source_dir / 'contrib' / 'packaging' / 'wix' / 'mercurial.wxs'
    defines['Version'] = version
    defines['Comments'] = 'Installs Mercurial version %s' % version
    defines['VCRedistSrcDir'] = str(hg_build_dir)

    run_candle(wix_path, build_dir, source, source_build_rel, defines=defines)

    msi_path = source_dir / 'dist' / (
        '%s-%s-%s.msi' % (msi_name, version, arch))

    args = [
        str(wix_path / 'light.exe'),
        '-nologo',
        '-ext', 'WixUIExtension',
        '-sw1076',
        '-spdb',
        '-o', str(msi_path),
    ]

    for source, rel_path in SUPPORT_WXS:
        assert source.endswith('.wxs')
        args.append(str(build_dir / ('%s.wixobj' % source[:-4])))

    args.append(str(build_dir / 'mercurial.wixobj'))

    subprocess.run(args, cwd=str(source_dir), check=True)

    print('%s created' % msi_path)

    return {
        'msi_path': msi_path,
    }


def build_signed_installer(source_dir: pathlib.Path, python_exe: pathlib.Path,
                           name: str, version=None, subject_name=None,
                           cert_path=None, cert_password=None,
                           timestamp_url=None):
    """Build an installer with signed executables."""

    post_build_fn = make_post_build_signing_fn(
        name,
        subject_name=subject_name,
        cert_path=cert_path,
        cert_password=cert_password,
        timestamp_url=timestamp_url)

    info = build_installer(source_dir, python_exe=python_exe,
                           msi_name=name.lower(), version=version,
                           post_build_fn=post_build_fn)

    description = '%s %s' % (name, version)

    sign_with_signtool(info['msi_path'], description,
                       subject_name=subject_name, cert_path=cert_path,
                       cert_password=cert_password, timestamp_url=timestamp_url)