diff: support diffing explicit files in subrepos Most other commands support implied recursion based on file names already.

fix: make the order of the work queue deterministic This makes any output generated during the parallel phase of execution stable if parallelism is disabled. This helps write tests like that in the future. Differential Revision: https://phab.mercurial-scm.org/D6166

fix: allow fixing untracked files when given as arguments It's more useful to fix the file than to silently avoid it when the user does this: hg fix --working-dir untracked-file We will still do nothing to ignored files, even if they are specified. This may be safer, given that such files can unexpectedly slip into the arguments via a shell glob or fileset. Differential Revision: https://phab.mercurial-scm.org/D6165

branchcache: have a hasnode function to validate nodes Upcoming patches will delay node validation until it's required. So we need to a way in branchcache class to check whether a node exists or node. Other options were making repo or changelog an attribute of branchcache object. But the branchcache depends on filters so I decided to pass a function object. Differential Revision: https://phab.mercurial-scm.org/D6157

branchcache: add attributes to track which nodes are verified Half of the cost of loading branchcache comes from verifiying all the nodes it has. We don't need to verify all the nodes in all the cases. Sometimes we need to verify only a set of nodes for a set of branches. We can ignore nodes of other branches as we are not going to read them. This patch introduces two attributes to branchcache class. _verifiedbranches is a set which will tell the branches for which it's head nodes are verified. _closedverified is a boolean which will tell whether all closednodes are verified or not. Another idea which I had was to keep a set of nodes which are verified, but it will be more ugly and difficult to track things on node level. Differential Revision: https://phab.mercurial-scm.org/D6156

branchcache: make entries a private attribute Differential Revision: https://phab.mercurial-scm.org/D6155

branchcache: introduce hasbranch() This will be used to check whether a branch exists or not. This will optimized in future. Differential Revision: https://phab.mercurial-scm.org/D6154

branchmap: drop branchcache.setdefault() (API) All the callers are updated to call setdefault of branchcache.entries Differential Revision: https://phab.mercurial-scm.org/D6153

branchcache: rename itervalues() to iterheads() The itervalues() exists because branchcache() had a dict interface. Since it no longer has a dict interface, it makes sense to have better function names. If a person does not understand how branchcache stores info, it will be hard for them to guess what itervalues() does. Differential Revision: https://phab.mercurial-scm.org/D6152

branchmap: remove the dict interface from the branchcache class (API) The current branchmap computation involves reading the whole branchmap from disk, validating all the nodes even if they are not required. This leads to a lot of time on repos which have large branchmap or a lot of branches. On large repos, this can validate around 1000's of nodes. On some operations, like finding whether a branch exists or not, we don't need to validate all the nodes. Or updating heads for a single branch. Before this patch, branchcache class was having dict interface and it was hard to keep track of reads. This patch removes the dict interface. Upcoming patches will implement lazy loading and validation of data and implement better API's. Differential Revision: https://phab.mercurial-scm.org/D6151

test-template: fix stdio mode on Windows Otherwise, CBOR data would be corrupted. Spotted by Matt Harbison.

samplehgrcs: update the list of suggested extensions Back in the day, this was color and pager, both of which are now default. Churn isn't that popular, but the other four below (obviously?) are.

samplehgrcs: clarify which lines should be uncommented The original wording has confused at least one person. Hopefully it's clearer this way. https://stackoverflow.com/questions/55288177/adding-hg-strip-to-hgrc-config-file

templatefilters: add {x|cbor} filter for custom CBOR output

template: add CBOR output format The whole output is wrapped as an array just like the other serialization formats. It's an indefinite-length array since the size is unknown while encoding. Maybe we can add 'cbor-stream' (and 'pickle-stream') as needed.

memfilectx: override copysource() instead of using dummy nodeid The "_copied" property in basefilectx is used by renamed() and copysource(). committablefilectx (which memfilectx subclasses) overrides renamed() and writes it in terms of copysource() instead of _copied. That means that the nodeid part of "_copied" is memfilectx is unused. Let's instead override copysource() too so we don't need the "_copied". Differential Revision: https://phab.mercurial-scm.org/D6159

memctx: rename constructor argument "copied" to "copysource" (API) It's just the path, not the nodeid, so "copysource" seems more appropriate. Differential Revision: https://phab.mercurial-scm.org/D6158

crecord: redraw the screen when starting up chunkselector Failure to do this can cause screen corruption like: <headerline> [X] filename.cc <several blank lines> <output from previous iteration of split that happened to be here> I believe this might only happen in some terminals, and maybe only when using the "alternate screen". Regardless of the exact conditions to reproduce, it should be safe to always clear it when starting up and is probably the correct thing to do anyway :) Differential Revision: https://phab.mercurial-scm.org/D6131

crecord: redraw the screen on ctrl-L This is the normal use of Ctrl-L, so I think this is going to be what most people expect it to do. We're keeping the adjustment of what line we're scrolled to as well. I believe both to be necessary to handle otherwise inescapable situations when we've got screen corruption or edge-cases during window resizing. Differential Revision: https://phab.mercurial-scm.org/D6130

crecord: completely redraw screen when coming back from editor Differential Revision: https://phab.mercurial-scm.org/D6129

tests: glob seconds in test-upgrade-repo.t I had the test failing locally for me with diff showing `1.4s` instead of 0.0s Differential Revision: https://phab.mercurial-scm.org/D6161

store: recommend using `hg debugrebuildfncache` is fncache is corrupted Differential Revision: https://phab.mercurial-scm.org/D6160

debugsparse: abort if the repository is not sparse instead of ui.status() This is similar to what narrow extension does. Differential Revision: https://phab.mercurial-scm.org/D6149

revert: option to choose what to keep, not what to discard I know the you (the reader) are probably tired of discussing how `hg revert -i -r .` should behave and so am I. And I know I'm one of the people who argued that showing the diff from the working copy to the parent was confusing. I think it is less confusing now that we show the diff from the parent to the working copy, but I still find it confusing. I think showing the diff of hunks to keep might make it easier to understand. So that's what this patch provides an option for. One argument doing it this way is that most people seem to find `hg split` natural. I suspect that is because it shows the forward diff (from parent commit to the commit) and asks you what to put in the first commit. I think the new "keep" mode for revert (this patch) matches that. In "keep" mode, all the changes are still selected by default. That means that `hg revert -i` followed by 'A' (keep all) (or 'c' in curses) will be different from `hg revert -a`. That's mostly because that was simplest. It can also be argued that it's safest. But it can also be argued that it should be consistent with `hg revert -a`. Note that in this mode, you can edit the hunks and it will do what you expect (e.g. add new lines to your file if you added a new lines when editing). The test case shows that that works. Differential Revision: https://phab.mercurial-scm.org/D6125

patch: include newline at EOF in help text for interactive patch The lack of a newline means that some "editors" that are useful in tests, such as `echo "+new line" >> "$1"` don't work. It's obviously easy to work around it, but newline at EOF seems like a good practice anyway. Differential Revision: https://phab.mercurial-scm.org/D6124

merge with stable

patch: include flag-only file changes in "special" when filtering (issue5864) This patch fix the issue5864 (or maybe issue5865 too) which occurs during split (or I should say at the time of filtering the hunks in interactive mode) where user hits a not ending loop of "no changes to record". And it's not only the case for split it will happen in every interactive case for e.g. `hg commit -i` or `hg uncommit -i` After looking into code I found that when filtering we have some notation called "special" for the file headers which doesn't contain any hunk and just contain the header (for e.g. newly added empty file or deleted file) where the user cannot change the content of operation. And I think we can put this "flag-only" file change in that same bucket of "special". But I doubt a bit about the case when a file have flag change and atleast one hunk then user won't be able to separate the flag change from hunks. Changed test file reflect the fixed behaviour. Differential Revision: https://phab.mercurial-scm.org/D6058

store: error out if fncache does not ends with a newline If fncache does not ends with a newline, chunk will not be fully consumed. It should be a bug somewhere or the fncache is corrupted if that happens. Let's error out in such cases. Differential Revision: https://phab.mercurial-scm.org/D6148

tracked: add documentation about `--import-rules` flag The documentation is inspired from the `--import-rules` flag of hg debugsparse command. Differential Revision: https://phab.mercurial-scm.org/D6150

discovery: fix embarrassing typo in slice definition The code introduced in e514799e4e07 ended up having a silly bug. The indexing selected a single item slice picking only p1. The discovery result was still correct, but the sampling was hampered, sometime leading to much more round trips being performed. Fixing this issue restore the previous sampling behavior. This fix has a negative performance impact on the pathological case the previous test has been built. # parent of this changesets ! wall 5.313884 comb 5.310000 user 5.260000 sys 0.050000 (best of 5) ! wall 6.711860 comb 6.710000 user 6.670000 sys 0.040000 (max of 5) ! wall 5.844016 comb 5.842000 user 5.784000 sys 0.058000 (avg of 5) ! wall 5.778635 comb 5.780000 user 5.740000 sys 0.040000 (median of 5) # With this changesets. ! wall 6.350879 comb 6.350000 user 6.300000 sys 0.050000 (best of 5) ! wall 6.653647 comb 6.660000 user 6.480000 sys 0.180000 (max of 5) ! wall 6.492762 comb 6.494000 user 6.414000 sys 0.080000 (avg of 5) ! wall 6.547577 comb 6.550000 user 6.490000 sys 0.060000 (median of 5) Changeset e514799e4e07 raised the question of using the "_uncheckedparentrevs" instead of the current code. So I ran comparative timing: # old code: 55919b96c02a (e514799e4e07 parent) ! wall 64.078708 comb 64.080000 user 63.160000 sys 0.920000 (best of 5) ! wall 68.296300 comb 68.290000 user 67.410000 sys 0.880000 (max of 5) ! wall 65.899075 comb 65.894000 user 65.082000 sys 0.812000 (avg of 5) ! wall 66.140286 comb 66.130000 user 65.330000 sys 0.800000 (median of 5) # buggy code: e514799e4e07 ! wall 46.605362 comb 46.610000 user 45.880000 sys 0.730000 (best of 5) ! wall 48.619659 comb 48.620000 user 47.890000 sys 0.730000 (max of 5) ! wall 47.350247 comb 47.350000 user 46.672000 sys 0.678000 (avg of 5) ! wall 46.983224 comb 46.980000 user 46.350000 sys 0.630000 (median of 5) # fixed code: e514799e4e07 with this fix ! wall 55.858460 comb 55.850000 user 55.090000 sys 0.760000 (best of 5) ! wall 59.048805 comb 59.060000 user 58.110000 sys 0.950000 (max of 5) ! wall 57.192639 comb 57.192000 user 56.350000 sys 0.842000 (avg of 5) ! wall 57.056373 comb 57.060000 user 56.160000 sys 0.900000 (median of 5) # version using uncheckedparents ! wall 56.471916 comb 56.470000 user 55.630000 sys 0.840000 (best of 5) ! wall 58.228793 comb 58.230000 user 57.600000 sys 0.630000 (max of 5) ! wall 57.377583 comb 57.378000 user 56.674000 sys 0.704000 (avg of 5) ! wall 57.008843 comb 57.010000 user 56.330000 sys 0.680000 (median of 5) So it looks like the overhead from `_uncheckedparentrevs` is not that impactful. I'll investigate this shortly. I'm almost done updating our benchmark suite with more meaningful discovery cases.

store: don't read the whole fncache in memory In large repositories with lot of files, the fncache grows more than 100 MB and reading that whole thing into memory slows things down. Let's not read the whole thing into memory. This patch changes fncache loading code to read 1 MB at once. Loading 1 MB at once saves ~1 sec on perffncacheload for our internal repository. I tried various values such as 0.5 MB, 5 MB, 10 MB but best results were produced using 1 MB as the chunksize. On a narrow clone with fncache around 40 MB, this patch saves ~0.04 seconds on average on perffncacheload. To test the code, I have coded an extension in test-fncache.t which set chunksize to 1 byte, and the test passes with that. Differential Revision: https://phab.mercurial-scm.org/D5296

wix: restore COPYING.rtf 8427fea04017 accidentally blew away the content of this file. As part of restoring the content, I updated the copyright year to 2019. Differential Revision: https://phab.mercurial-scm.org/D6098

store: move logic to check for invalid entry in fncache to own function This helps separate the original reading logic from the one which finds for an invalid entry. Differential Revision: https://phab.mercurial-scm.org/D6030

py3: add test-phabricator.py to python3-whitelist Differential Revision: https://phab.mercurial-scm.org/D6114

py3: convert to/from bytes/unicode for json.(dump|load)s in debugcallconduit Differential Revision: https://phab.mercurial-scm.org/D6113

py3: use pycompat.byteskwargs on opts in phabricator.py Differential Revision: https://phab.mercurial-scm.org/D6107

watchman: ignore some of watchman errors Don't display 'illegal_fstypes' errors. In environments with network filesystems, the error messages are quickly pilling up and polluting outputs. Differential Revision: https://phab.mercurial-scm.org/D5955

watchman: add the possibility to set the exact watchman binary location This is necessary to make rolling releases of new watchman versions across users. Differential Revision: https://phab.mercurial-scm.org/D5954

context: use wdirhex constant instead of calculating it Differential Revision: https://phab.mercurial-scm.org/D6143

split: use the new movedirstate() we now have in scmutil This avoids unnecessarily touching the working copy when splitting the parent of the working copy. That also makes the test-removeemptydirs.t case invalid, so we can just delete it. Differential Revision: https://phab.mercurial-scm.org/D6127

manifestcache: use `wcache` directory for manifest cache The manifest full text cache is tightly related to the working copy. We should use the `wcache` directory for it, instead of the `cache`. Otherwise, multiple shares would keep overwriting each other cache entry and we loose its benefit. This is also more consistent with the fact this cache file is protected by `wlock`.

manifestcache: protect write with `wlock` instead of `lock` The `wlock` is taken by both `update` and `commit` type operation. This would help persisting the cache more aggressively. An explicit test is introduced. However, we can already see the effect of this change on earlier test output.

manifestcache: clear the cache before testing the debug command Right now the cache is empty before this section of the test. However, we are about to improve the persistence of the cache (putting it under `wllock`, intead of `lock`). So we install the cleanup before.

manifestcache: abstract the filename in a class attribute This make the code clearer and simpler to update.

manifestcache: skip setup earlier if we don't have the lock There a no point preparing a closure if we are not going to use it.

manifestcache: test the cache is warm after a commit

manifestcache: stop altering the lru cache order while displaying it Accessing value with `.get` alter the iteration order and make the output of the debug command misbehave, showing multiple entry twice. We need more than 2 entry to see the bug, so there are not test change. Later test will introduce a third entry and would fail without this fix.

manifestcache: support multiple cache addition in one debug command run This is more practical.

wix: autogenerate wxs file for library files Currently, dist.wxs contains an enumeration of .pyd and .dll files staged to dist/lib by py2exe. Having a manual list of files is error prone, as things can easily get out of sync (as the previous commit demonstrates). This is especially an issue for TortoiseHG, which ships a number of custom modules, which may pull in additional dependencies. Let's prevent this problem from manifesting by dynamically generating a .wxs file containing the .pyd and .dll files staged by py2exe. Differential Revision: https://phab.mercurial-scm.org/D6139

wix: introduce variable to hold path to wix packaging directory For convenience. Differential Revision: https://phab.mercurial-scm.org/D6138

wix: package missing .dll and .pyd files dist.wxs is currently missing some .pyd and .dll files which are picked up and staged by py2exe. This means that the WiX installer is missing some Python extension modules and their dependencies which are referenced by Mercurial or a Python package distributed with it. This commit adds the missing files to the WiX installer. Differential Revision: https://phab.mercurial-scm.org/D6137

setup: exclude crypt32.dll in py2exe builds py2exe is picking up crypt32.dll as a dependency and is including the DLL in the dist/lib directory, where it can get picked up by an installer and distributed. crypt32.dll is a core Windows DLL since Windows XP. We don't need to distribute it. Differential Revision: https://phab.mercurial-scm.org/D6136

packaging: don't bundle DLLs in py2exe library.zip for x86 builds I had ported the x86/x64 behavior difference from the Inno Setup installer files. Why things were this way, I'm not sure. The WiX configuration files are expecting to have standalone DLL files for both configurations. And the 32-bit WiX installers were broken due to missing DLLs. Let's standardize on standalone DLL files on all configurations for consistency. I /think/ this will be faster, as I /think/ py2exe binaries would have to extract the DLL to a temporary file in order to load it. But I'm not 100% sure about that. Differential Revision: https://phab.mercurial-scm.org/D6135

packaging: convert files to LF My editor accidentally wrote CRLF line endings because I authored this code on Windows. Derp. I'm kinda surprised no linters caught this... # no-check-commit to avoid false positive for foo_bar names Differential Revision: https://phab.mercurial-scm.org/D6134

dirstate: remove obsolete reference to dirstate.beginparentchange The only valid API since 265e91da56fd (dirstate: drop deprecated methods (API), 2018-02-02) is the context manager returned from dirstate.parentchange(). Differential Revision: https://phab.mercurial-scm.org/D6126

py3: use pycompat.iterbytestr to convert memoryview slice to bytestring Otherwise ch is the int value of the byte in py3 rather than the actual character. Differential Revision: https://phab.mercurial-scm.org/D6103

rebase: fix crash with in-memory rebase and copies When using regular on-disk rebase, filectx.markcopies() calls to dirstate.copy(), which happily records the copy. Then it's simply ignored if it doesn't matter for the commit (as in the test case I added in the previous patch). Let's do the same for overlayworkingctx. Differential Revision: https://phab.mercurial-scm.org/D6133

test: demonstrate crash with in-memory rebase and copies In the added test case, there is a merge commit that has one obsolete parent with a rename. Since the rename is not in the other parent, pathcopies() from that other parent will include the copy. Then when we try to rebase this merge commit onto another commit that has the same content changes, but no tracking of the rename (because it was done with "hg remove; hg add" instead of "hg mv"), we try to propagate the copy information. That fails because overlayworkingctx expects a file to be modified if it's going to have copy information. Differential Revision: https://phab.mercurial-scm.org/D6132

manifestcache: actually honor --clear Before this change, the --clear flag was not clearing the on disk cache. (We also remove the extra verbosity when using --clear. Same as what we did for --add)

manifestcache: make sure the entry are ordered by access time This is an LRU cache, let us make sure of that.

manifestcache: adding a second distinct entry Let makes sure the cache can hold multiple value.

manifestcache: test that adding the same entry twice do not duplicates it Simple sanity check.

manifestcache: do not display data when using --add If the command invocation is about adding a new entry, we should remain terse (the same as we do for many commands).

manifestcache: only lock the repository if the debug command touch the cache Not doing so had two consequences: 1) the command cannot be run on read only repositories, 2) when using --add on an empty cache, the command crash prematurely trying to read the cache file on disk.

manifestcache: further fix to debug command output Removing more capital letters. The output will get a test once other issues get fixed.

manifestcache: test and fix some output of the debug command The message was lacking an end of line. In addition we do not capitalize output in Mercurial.

chunkselector: fix typos in instructions when user reviews patch Differential Revision: https://phab.mercurial-scm.org/D6121

scmutil: document matcher argument of movedirstate() Differential Revision: https://phab.mercurial-scm.org/D6120

uncommit: move _movedirstate() to scmutil for reuse The function should be applicable generically when moving from one commit to another. I'll try to add more callers when I find time. I'm not convinced it's handling all the cases correctly, but we should have a generic function for this kind of operation, so I think it belongs somewhere in core (not in the uncommit extension). Differential Revision: https://phab.mercurial-scm.org/D6119

copies: remove dependency on scmutil by directly using match.exact() I want to add a dependency from scmutil.copies(), so I need to remove this dependency first. Differential Revision: https://phab.mercurial-scm.org/D6118

uncommit: convert _fixdirstate() into _movedirstate() _fixdirstate() already also updates to the given commit, so let's rename it to _movedirstate(). Also update the documentation and drop the unnecessary "curctx" argument, since that should always be repo['.']. Differential Revision: https://phab.mercurial-scm.org/D6117

updatecaches: also warm the tags caches Resolving any name requires the tags cache to be warm. We make sure that `hg debugupdatecache` warm the tag cache entry too.

updatecaches: also warm revbranchcache for filtered revisions We are in the "full" case, so we better warm everything we can.

copies: handle a case when both merging csets are not descendant of merge base This patch fix the behaviour of fullcopytracing algorithm in the case when both the merging csets are not the descendant of merge base. Although it seems to be the rare case when both the csets are not descendant of merge base. But it can be seen in most of cases of content-divergence in evolve extension, where merge base is the common predecessor. Previous patch added a test where this algorithm can fail to continue because of an assumption that only one of the two csets can be dirty. This patch fix that error. For refrence I suggest you to look into the previous discussion held on a patch sent by Pulkit: https://phab.mercurial-scm.org/D3896 Differential Revision: https://phab.mercurial-scm.org/D5963

copies: add test that makes both the merging csets dirty and fails This patch is a part of series which is about the case when both the merging csets are not descendant of merge base. The existing code assumes if c1 is dirty there shouldn't be any partial copies from c2 i.e both2['incomplete'] and same for c2, if c2 is dirty both1['incomplete'] should be empty, but this is not the right assumption. Now as we know we can have both c1 and c2 dirty at the same time, it is possible that c1 is dirty and both2['incomplete'] has some value. Or if c2 is dirty and both1['incomplete'] has some value. Added test shows that because of this assumption it could fail. Differential Revision: https://phab.mercurial-scm.org/D5962

copies: add test that makes both the merging csets dirty and run w/o error This series of patches is to cover a case in fullcopytracing algorithms where both the merging csets are not descendant of merge base. In this algorithm we call a merging cset "dirty" if that cset is not the descendant of merge base. That said, added test in this patch cover case when both the merging csets are "dirty". Actually this case of "both dirty" was encountered by Pulkit when he was working on content-divergence where it is possible that both the csets are not descendant of merging base. For reference you can look into: https://phab.mercurial-scm.org/D3896 As this test run fine without any error and correctly traced the copies, I added this test to make sure that it doesn't break even after I will modify some code in next patches to fix an error. Next patch adds the tests where this algorithm throws an error for the same case of "both dirty". Differential Revision: https://phab.mercurial-scm.org/D5961

tests: stabilize test-bundle.t on Windows Similar to 92055d539e49.

discovery-helper: use reflink copy if available A reflink copy will copy the files "as usual" but keep using the same data block underneath. This is only supported by "copy on write" file system like btrfs or zfs. This will achieve similar performance that the existing hardlink clone that Mercurial performs with the same initial space saving. However, it will behave better on revlogs start being touch by strip. Instead of duplicating all data in the touched revlogs, only the block actually affected by the strip will be duplicated. This save a lot of space when building many variants of large repositories. The --reflink=always flag make sure the `cp` call fails if reflink copies are not supported. Falling back to local clone.

discovery-helper: bail out if destination already exists

discovery-helper: move repository creation in a function This makes it easier to update this duplicated code. (we do a small output fix as we go)

discovery-helper: add an extra argument to generate only one repo This is useful to generate left and right in parallel when dealing with very large repositories.

wix: remove enum and future packages These were cargo culted from the THG installer code. I'm not sure what needs them in THG land. But the official MSIs certainly do not - at least not as direct dependencies. .. bc:: The Windows MSI installers no longer include the enum and future Python packages. Differential Revision: https://phab.mercurial-scm.org/D6101

wix: remove pywin32 This dependency was for ancient Mercurial versions. We recently removed it from the Inno Setup installers. So let's remove it from the WiX installers as well. .. bc:: The Windows MSI installers no longer include the pywin32 Python package. Differential Revision: https://phab.mercurial-scm.org/D6100

wix: remove sphinx and dependencies Sphinx was cargo culted into our install environment as part of emulating TortoiseHG's behavior. THG seems to install Sphinx in order to generate THG specific documentation. We don't appear to need Sphinx or any of its dependencies in the official WiX installers. So remove it. This shaves ~1MB off the size of the MSI installers. .. bc:: The Windows MSI installers no longer include the Python sphinx package and its various dependencies. Differential Revision: https://phab.mercurial-scm.org/D6099

wix: functionality to automate building WiX installers Like we did for Inno Setup, we want to make it easier to produce WiX installers. This commit does that. We introduce a new hgpackaging.wix module for performing all the high-level tasks required to produce WiX installers. This required miscellaneous enhancements to existing code in hgpackaging, including support for signing binaries. A new build.py script for calling into the module APIs has been created. It behaves very similarly to the Inno Setup build.py script. Unlike Inno Setup, we didn't have code in the repo previously to generate WiX installers. It appears that all existing automation for building WiX installers lives in the https://bitbucket.org/tortoisehg/thg-winbuild repository - most notably in its setup.py file. My strategy for inventing the code in this commit was to step through the code in that repo's setup.py and observe what it was doing. Despite the length of setup.py in that repository, the actual amount of steps required to produce a WiX installer is actually quite low. It consists of a basic py2exe build plus invocations of candle.exe and light.exe to produce the MSI. One rabbit hole that gave me fits was locating the Visual Studio 9 C Runtime merge modules. These merge modules are only present on your system if you have a full Visual Studio 2008 installation. Fortunately, I have a copy of Visual Studio 2008 and was able to install all the required updates. I then uploaded these merge modules to a personal repository on GitHub. That is where the added code references them from. We probably don't need to ship the merge modules. But that is for another day. The installs from the MSIs produced with the new automation differ from the last official MSI in the following ways: * Our HTML manual pages have UNIX line endings instead of Windows. * We ship modules in the mercurial.pure package. It appears the upstream packaging code is not including this package due to omission (they supply an explicit list of packages that has drifted out of sync with our setup.py). * We do not ship various distutils.* modules. This is because virtualenvs have a custom distutils/__init__.py that automagically imports distutils from its original location and py2exe gets confused by this. We don't use distutils in core Mercurial and don't provide a usable python.exe, so this omission should be acceptable. * The version of the enum package is different and we ship an enum.pyc instead of an enum/__init__.py. * The version of the docutils package is different and we ship a different set of files. * The version of Sphinx is drastically newer and we ship a number of files the old version did not. (I'm not sure why we ship Sphinx - I think it is a side-effect of the way the THG code was installing dependencies.) * We ship the idna package (dependent of requests which is a dependency of newer versions of Sphinx). * The version of imagesize is different and we ship an imagesize.pyc instead of an imagesize/__init__.pyc. * The version of the jinja2 package is different and the sets of files differs. * We ship the packaging package, which is a dependency for Sphinx. * The version of the pygments package is different and the sets of files differs. * We ship the requests package, which is a dependency for Sphinx. * We ship the snowballstemmer package, which is a dependency for Sphinx. * We ship the urllib3 package, which is a dependency for requests, which is a dependency for Sphinx. * We ship a newer version of the futures package, which includes a handful of extra modules that match Python 3 module names. # no-check-commit because foo_bar naming Differential Revision: https://phab.mercurial-scm.org/D6097

wix: move contrib/wix to contrib/packaging/wix We're trying to consolidate all our packaging code into contrib/packaging. Let's move the WiX files there. Differential Revision: https://phab.mercurial-scm.org/D6096

wix: remove hg.cmd This file is not referenced anywhere AFAICT. Differential Revision: https://phab.mercurial-scm.org/D6095

setup: include hgext3rd package in py2exe builds This is a core Mercurial package and we should always ship it. Differential Revision: https://phab.mercurial-scm.org/D6094

setup: properly install build_hgextindex for py2exe builds Because the hgbuild class has a private copy of build.sub_commands, modifying build.sub_commands from this code effectively resulted in a no-op. Registering the sub-command on hgbuild actually results in the sub-command running when building Mercurial. Differential Revision: https://phab.mercurial-scm.org/D6093

setup: configure py2exe config via environment variables The Inno Setup and WiX installers ship a different set of packages with py2exe builds. And there are multiple WiX installer variants (e.g. TortoiseHG). Since there are multiple variants of py2exe configs and they can be defined by entities not in our repository, let's provide a mechanism for setup.py to supplement behavior via environment variables. This is slighly less hacky than a setup.cfg file IMO since the caller doesn't need to worry about mutating global state of the source directory. Differential Revision: https://phab.mercurial-scm.org/D6092

packaging: extract py2exe functionality to own module py2exe builds are shared between Inno Setup and WIX. We'll want the logic for performing py2exe builds to be reusable across the code for both installers. This commit extracts the py2exe-specific functionality into its own module. There's definitely room to customize things further. This will be done in future commits, as necessary. (I'm not even sure what customizations WIX will require yet. Presumably a lot.) Differential Revision: https://phab.mercurial-scm.org/D6091

packaging: extract python exe info to own function This is generic functionality. We'll need it for WIX. As part of the port, we expose the full version and return the data as a dict. Differential Revision: https://phab.mercurial-scm.org/D6090

packaging: don't use temporary directory We were no longer doing anything with it after extracting virtualenv and py2exe to the build directory. Differential Revision: https://phab.mercurial-scm.org/D6089

packaging: extract virtualenv and py2exe to build directory The build directory is essentially a cache. We can extract the virtualenv and py2exe package sources to this directory. Differential Revision: https://phab.mercurial-scm.org/D6088

packaging: move Inno Setup core logic into a module Aspects of building the Inno Setup and WIX installers are shared. It will make sense for them to share code. Plus, having code in a reusable library (as opposed to a standalone script) is just a better approach. This commit moves the core logic to build the Inno Setup installer into the hgpackaging package. inno/build.py is now a simple frontend script that calls into a module to do the bulk of the work. As part of this change, I also found a typo in build() where it was referencing "iscc" instead of "iscc_exe." Because "iscc" was in the global scope via the only caller, things just happened to work before. Another benefit of always using functions and not putting global code for __main__ in the same file as library code. Differential Revision: https://phab.mercurial-scm.org/D6087

packaging: move find_vc_runtime_files() into hgpackaging.util In preparation for moving the bulk of the Inno Setup code into hgpackaging. Differential Revision: https://phab.mercurial-scm.org/D6086