wix: remove sphinx and dependencies Sphinx was cargo culted into our install environment as part of emulating TortoiseHG's behavior. THG seems to install Sphinx in order to generate THG specific documentation. We don't appear to need Sphinx or any of its dependencies in the official WiX installers. So remove it. This shaves ~1MB off the size of the MSI installers. .. bc:: The Windows MSI installers no longer include the Python sphinx package and its various dependencies. Differential Revision: https://phab.mercurial-scm.org/D6099

wix: functionality to automate building WiX installers Like we did for Inno Setup, we want to make it easier to produce WiX installers. This commit does that. We introduce a new hgpackaging.wix module for performing all the high-level tasks required to produce WiX installers. This required miscellaneous enhancements to existing code in hgpackaging, including support for signing binaries. A new build.py script for calling into the module APIs has been created. It behaves very similarly to the Inno Setup build.py script. Unlike Inno Setup, we didn't have code in the repo previously to generate WiX installers. It appears that all existing automation for building WiX installers lives in the https://bitbucket.org/tortoisehg/thg-winbuild repository - most notably in its setup.py file. My strategy for inventing the code in this commit was to step through the code in that repo's setup.py and observe what it was doing. Despite the length of setup.py in that repository, the actual amount of steps required to produce a WiX installer is actually quite low. It consists of a basic py2exe build plus invocations of candle.exe and light.exe to produce the MSI. One rabbit hole that gave me fits was locating the Visual Studio 9 C Runtime merge modules. These merge modules are only present on your system if you have a full Visual Studio 2008 installation. Fortunately, I have a copy of Visual Studio 2008 and was able to install all the required updates. I then uploaded these merge modules to a personal repository on GitHub. That is where the added code references them from. We probably don't need to ship the merge modules. But that is for another day. The installs from the MSIs produced with the new automation differ from the last official MSI in the following ways: * Our HTML manual pages have UNIX line endings instead of Windows. * We ship modules in the mercurial.pure package. It appears the upstream packaging code is not including this package due to omission (they supply an explicit list of packages that has drifted out of sync with our setup.py). * We do not ship various distutils.* modules. This is because virtualenvs have a custom distutils/__init__.py that automagically imports distutils from its original location and py2exe gets confused by this. We don't use distutils in core Mercurial and don't provide a usable python.exe, so this omission should be acceptable. * The version of the enum package is different and we ship an enum.pyc instead of an enum/__init__.py. * The version of the docutils package is different and we ship a different set of files. * The version of Sphinx is drastically newer and we ship a number of files the old version did not. (I'm not sure why we ship Sphinx - I think it is a side-effect of the way the THG code was installing dependencies.) * We ship the idna package (dependent of requests which is a dependency of newer versions of Sphinx). * The version of imagesize is different and we ship an imagesize.pyc instead of an imagesize/__init__.pyc. * The version of the jinja2 package is different and the sets of files differs. * We ship the packaging package, which is a dependency for Sphinx. * The version of the pygments package is different and the sets of files differs. * We ship the requests package, which is a dependency for Sphinx. * We ship the snowballstemmer package, which is a dependency for Sphinx. * We ship the urllib3 package, which is a dependency for requests, which is a dependency for Sphinx. * We ship a newer version of the futures package, which includes a handful of extra modules that match Python 3 module names. # no-check-commit because foo_bar naming Differential Revision: https://phab.mercurial-scm.org/D6097

wix: move contrib/wix to contrib/packaging/wix We're trying to consolidate all our packaging code into contrib/packaging. Let's move the WiX files there. Differential Revision: https://phab.mercurial-scm.org/D6096

wix: remove hg.cmd This file is not referenced anywhere AFAICT. Differential Revision: https://phab.mercurial-scm.org/D6095

setup: include hgext3rd package in py2exe builds This is a core Mercurial package and we should always ship it. Differential Revision: https://phab.mercurial-scm.org/D6094

setup: properly install build_hgextindex for py2exe builds Because the hgbuild class has a private copy of build.sub_commands, modifying build.sub_commands from this code effectively resulted in a no-op. Registering the sub-command on hgbuild actually results in the sub-command running when building Mercurial. Differential Revision: https://phab.mercurial-scm.org/D6093

setup: configure py2exe config via environment variables The Inno Setup and WiX installers ship a different set of packages with py2exe builds. And there are multiple WiX installer variants (e.g. TortoiseHG). Since there are multiple variants of py2exe configs and they can be defined by entities not in our repository, let's provide a mechanism for setup.py to supplement behavior via environment variables. This is slighly less hacky than a setup.cfg file IMO since the caller doesn't need to worry about mutating global state of the source directory. Differential Revision: https://phab.mercurial-scm.org/D6092

packaging: extract py2exe functionality to own module py2exe builds are shared between Inno Setup and WIX. We'll want the logic for performing py2exe builds to be reusable across the code for both installers. This commit extracts the py2exe-specific functionality into its own module. There's definitely room to customize things further. This will be done in future commits, as necessary. (I'm not even sure what customizations WIX will require yet. Presumably a lot.) Differential Revision: https://phab.mercurial-scm.org/D6091

packaging: extract python exe info to own function This is generic functionality. We'll need it for WIX. As part of the port, we expose the full version and return the data as a dict. Differential Revision: https://phab.mercurial-scm.org/D6090

packaging: don't use temporary directory We were no longer doing anything with it after extracting virtualenv and py2exe to the build directory. Differential Revision: https://phab.mercurial-scm.org/D6089

packaging: extract virtualenv and py2exe to build directory The build directory is essentially a cache. We can extract the virtualenv and py2exe package sources to this directory. Differential Revision: https://phab.mercurial-scm.org/D6088

packaging: move Inno Setup core logic into a module Aspects of building the Inno Setup and WIX installers are shared. It will make sense for them to share code. Plus, having code in a reusable library (as opposed to a standalone script) is just a better approach. This commit moves the core logic to build the Inno Setup installer into the hgpackaging package. inno/build.py is now a simple frontend script that calls into a module to do the bulk of the work. As part of this change, I also found a typo in build() where it was referencing "iscc" instead of "iscc_exe." Because "iscc" was in the global scope via the only caller, things just happened to work before. Another benefit of always using functions and not putting global code for __main__ in the same file as library code. Differential Revision: https://phab.mercurial-scm.org/D6087

packaging: move find_vc_runtime_files() into hgpackaging.util In preparation for moving the bulk of the Inno Setup code into hgpackaging. Differential Revision: https://phab.mercurial-scm.org/D6086

packaging: move DOWNLOADS dict to hgpackaging.downloads We'll want to keep state in sync between multiple packaging tools. It makes sense to share a central data structure defining downloads. We also change the function to return the downloads entry so callers don't have to access the global DOWNLOADS in the new location. Differential Revision: https://phab.mercurial-scm.org/D6085

packaging: split downloading code into own module As we will introduce more code to support packaging, it will be useful to have download code in its own module. Differential Revision: https://phab.mercurial-scm.org/D6084

packaging: establish hgpackaging package Previously, contrib/packaging behaved as a root to a package directory and we had a "packagingutil" module. As I work more on packaging code, we'll want to have more code shared between different packaging tools. I think it makes sense to have a single package containing multiple modules than multiple top-level modules. This commit establishes an "hgpackaging" package by moving the existing packagingutil code to it. Differential Revision: https://phab.mercurial-scm.org/D6083

py3: use % instead of .format() on a bytestring Differential Revision: https://phab.mercurial-scm.org/D6112

py3: use r'' for group name arguments to MatchObjects in phabricator.py MatchObject group names are strings, not bytes. Using bytes in py3 leads to an IndexError. Differential Revision: https://phab.mercurial-scm.org/D6111

py3: use %d instead of %s when formatting an int into a byte string Differential Revision: https://phab.mercurial-scm.org/D6110

py3: only pass unicode to json.dumps in writediffproperties Differential Revision: https://phab.mercurial-scm.org/D6109

py3: fix a few "dict keys as str instead of bytes" issues in phabricator.py Differential Revision: https://phab.mercurial-scm.org/D6108

py3: convert URL to str before passing it to request Differential Revision: https://phab.mercurial-scm.org/D6106

py3: convert indexes into bytes when enumerating lists in urlencodenested Otherwise it'll try to insert them them into a %s slot in a b'' later and fail. Differential Revision: https://phab.mercurial-scm.org/D6105

py3: don't try and format a bare dict into a byte string in callconduit Differential Revision: https://phab.mercurial-scm.org/D6104

py3: use fsencode for vcr recording paths and strings for custom_patches args This fixes phabricator.py's vcrcommand under py3 Differential Revision: https://phab.mercurial-scm.org/D6102

phabricator: convert conduit response JSON unicode to bytes inside callconduit Previously the byte conversion was happening piecemeal in callers, and in the case of createdifferentialrevision not at all, leading to UnicodeEncodeErrors when trying to phabsend a commit with a description containing characters not representable in ascii. (issue6040) Remove all the scattered encoding.unitolocal calls and perform it once, inside callconduit, on the entire response dict recursively before returning it, in keeping with the strategy of converting at the earliest opportunity. Convert all keys used on returned object to bytes. Modify the phabsend tests to test this by adding a € to the commit message of alpha. Differential Revision: https://phab.mercurial-scm.org/D6044

transaction: include txnname in the hookargs dictionary There is no reason to not include the txnname alongside the txnid in all case. The python hooks already have them, so aligning the the shell hooks seems it could be useful in the future. (I don't have a strong opinion about this, we can also decide to never align the python and shell hooks and this and I'll drop this patch).

discovery-helper: reflect argument value in the name of the results It is common to create multiple pairs of repositories using different argument values. Recording the argument value in the results names has two main advantages: * It is easy to remember the value used to create a pair, * It get simpler to create multiple pair at the same time from the same source. Previously, running: `./discovery-helper.sh pypy 50 10` would create a `pypy-left` and `pypy-right` repository. Now it will create `pypy-50h-10d-left` and `pypy-50h-10d-right`.

discovery-helper: echo the stripped revsets early Having them printed early make it easy for a user to just grab the generated revset and directly uses them

contrib: move the `discovery-helper.sh` script in`perf-utils` directory The script appeared before the directory. However the directory exists for this kind of script. We move it there.