mail: add type hints for pytype We essentially annotate functions in which handling of bytes/str is not obvious in order to hopefully clear things out. See also changeset 2ade00f3b03b introducing typing hints in Mercurial. Most types are straightforward but a few is wrong, and we need to either disable pytype on respective instructions or use wrong annotations. These will be fixed in next changesets. Notice the type Union[bytes, str] of "s" parameter of headencode(), this reflects how email.header.Header.append() behaves.

mail: fix a bad return type in _encode() This particular instruction returned only a string and omitted the charset value.

packaging: stage files and dynamically generate WiX installer Like we did for Inno, we want to make the WiX installer "dumb" and simply consume source files from a directory tree rather than have to define every single file in installer files. This will greatly decrease the amount of effort required to maintain the WiX installer since we don't have to think that much about keeping files in sync. This commit changes the WiX packager to populate a staging directory as part of packaging. After it does so, it scans that directory and dynamically generates WiX XML defining the content within. The IDs and GUIDs being generated are deterministic. So, upgrades should work as expected in Windows Installer land. (WiX has a "heat" tool that can generate XML by walking the filesystem but it doesn't have this deterministic property, sadly.) As part of this change, GUIDs are now effectively reset. So the next upgrade should be a complete wipe and replace. This could potentially cause issues. But in my local testing, I was able to upgrade an existing 5.1.2 install without issue. Compared to the previous commit, the installed files differ in the following: * A ReleaseNotes.txt file is now included * A hgrc.d/editor.rc file is now generated (mercurial.rc has been updated to reflect this logical change to the content source) * All files are marked as read-only. Previously, only a subset of files were. This should help prevent unwanted tampering. Although we may want to consider use cases like modifying template files... This change also means that Inno and WiX are now using very similar code for managing the install layout. This means that on disk both packages are nearly identical. The differences in install layout are as follows: * Inno has a Copying.txt vs a COPYING.rtf for WiX. (The WiX installer wants to use RTF.) * Inno has a Mercurial.url file that is an internet shortcut to www.mercurial-scm.org. (This could potentially be removed.) * Inno includes msvc[mpr]90.dll files and WiX does not. (WiX installs the MSVC runtime via merge modules.) * Inno includes unins000.{dat,exe} files. (WiX's state is managed by Windows Installer, which places things elsewhere.) Because file lists are dynamically generated now, the test ensuring things remain in sync has been deleted. Good riddance. While this is a huge step towards unifying the Windows installers, there's still some improvements that can be made. But I think it is worth celebrating the milestone of getting both Inno and WiX to essentially share core packaging code and workflows. That should make it much easier to change the installers going forward. This will aid support of Python 3. Differential Revision: https://phab.mercurial-scm.org/D7173

tests: add optional output when wheel is missing If the `wheel` package isn't installed, pip will emit a warning on stderr complaining about that. Alternatively, we could redirect stderr to stdout and swallow the warning. But I like tests having visibility of warnings, as it helps keep us honest regarding edge cases. Differential Revision: https://phab.mercurial-scm.org/D7371

py3: replace "%r" by"'%s'% for py3-compatible (and clearer) quoting in chg Differential Revision: https://phab.mercurial-scm.org/D7375

dirstate: respect request to not list unknown/ignored/clean files (API) Unknown files that are explicitly mentioned by the matcher are returned even if the caller said unknown=False (and it seems the same is done for ignored files). That seems pretty surprising. Let's make the interface less surprising by respecting the caller's request. Differential Revision: https://phab.mercurial-scm.org/D7150

py3: pass a bytes value for "msg" to nouideprecwarn() That function formats "msg" with the "version" value. On Python 3, this leads to "TypeError: can only concatenate str (not "bytes") to str". Also eliminate spurious strings concatenation in single-line declarations. Differential Revision: https://phab.mercurial-scm.org/D7373

remotefilelog: handle **kwargs correctly when overriding changelog.add() Differential Revision: https://phab.mercurial-scm.org/D7372

packaging: ship all help .txt files on WiX These are technically not needed. But it is easier to ship all files than to cherry-pick. A `make install` will copy these files, so the new behavior is consistent with that. This also makes WiX consistent with Inno, which is my main reason for doing this. If we don't want to ship the files (which is a valid argument), I think we can do that in a follow up. Differential Revision: https://phab.mercurial-scm.org/D7166

packaging: install contrib/mq.el on WiX This makes things consistent with the Inno installer. Differential Revision: https://phab.mercurial-scm.org/D7167

fuzz: clean out most of fuzzutil It's now a header-only setup that just selects absl:: or std:: versions of things as needed, and a logging helper. There's some room for future cleanups here: we could move to just requiring a C++17 compiler and get rid of the absl stuff. Also, the mpatch parser has a fair amount of parsing the input string into char* blocks that we can and probably should fix that up to use FuzzedDataProvider as well. Differential Revision: https://phab.mercurial-scm.org/D7370

fuzz: migrate xdiff fuzzer to use FuzzedDataProvider Differential Revision: https://phab.mercurial-scm.org/D7369

fuzz: migrate bdiff fuzzer to use FuzzedDataProvider Differential Revision: https://phab.mercurial-scm.org/D7368

dirstate: re-blacken file This got overlooked in landing a recent patch, but it's easy to fix up. Differential Revision: https://phab.mercurial-scm.org/D7367

rust-matchers: remove default implementations for `Matcher` trait We don't expect a whole lot of matchers to be defined, and this makes it more obvious what a matcher does by reading its `impl Matcher for FooMatcher`. This patch has the added benefit of fixing the `AlwaysMatcher`, its `matches` function differs from the former default. Differential Revision: https://phab.mercurial-scm.org/D7255

packaging: use lower case for RC files The file case doesn't matter. I'm not sure why it is capitalized. This is a source of inconsistency between Inno and WiX. So let's standardize on the lower case names. The MSI should remove the old files before installing the new one, so there shouldn't be a problem with a case collision. And, users should not have modified these files, so it should be safe to rename them. Differential Revision: https://phab.mercurial-scm.org/D7172

packaging: write out editor.rc in Python Let's dynamically generate the file from Python instead of relying on Inno functionality, as that will give more visibility into what's going on. I spotted this as part of converging the install layout of Inno and WiX: the former doesn't install an editor.rc and instead defines the editor via its mercurial.rc file. We will change the behavior of WiX later once it is consuming the staged install directory for packaging. Differential Revision: https://phab.mercurial-scm.org/D7171

packaging: stop installing Contributors.txt This file isn't up to date. If we are going to call out contributors, we should auto generate a file from `hg log` or something. I don't think we need to include it in the Windows install. This change makes Inno more consistent with WiX, which doesn't ship this file. Differential Revision: https://phab.mercurial-scm.org/D7170

packaging: remove hg-ssh.8.html from Inno installer We don't ship hg-ssh because it requires a python.exe to run, which we don't ship. So it doesn't make sense to ship the HTML documentation for this tool. This change makes the Inno install layout more consistent with WiX, which doesn't ship this file. Functionality for removing files has been made generic, in anticipation of future expansion. Differential Revision: https://phab.mercurial-scm.org/D7169

packaging: install .rc files to hgrc.d Currently, Inno installs these .rc files to a default.d directory and WiX to a hgrc.d directory. Mercurial automatically loads files in an hgrc.d directory next to the executable and in an "internal" default.d directory. This "internal" directory is util.datadir, which on Windows installs made with py2exe is the directory containing hg.exe. So on Windows both hgrc.d and default.d directories relative to hg.exe are loaded. For the install layout, I think it makes sense for hgrc.d to be used, as that path is documented as non-internal. So this commit changes the Inno install layout to use hgrc.d instead of default.d. .. bc:: Changed default config file location in Windows .exe installers Mercurial's installer installs some default .rc config files. Previously, these were installed in a default.d directory. These are now installed in an hgrc.d directory. Custom default.d/*.rc files should still be loaded. But it is recommended to move these to hgrc.d/*.rc. This change only affects the .exe installers (not the .msi installers). Differential Revision: https://phab.mercurial-scm.org/D7168

rust: introduce SIZE_FROM_OTHER_PARENT constant This will clarify what `-2` meant in the codebase. This change also merged imports in affected files for cleanup. Differential Revision: https://phab.mercurial-scm.org/D7301

rust-status: refactor dispatch case for normal files This should make the code easier to read and more idiomatic. Differential Revision: https://phab.mercurial-scm.org/D7300

rust-status: return a ParallelIterator instead of a Vec from stat_dmap_entries This allows the caller function to choose when and how the iteration and/or collection happens. This change also cleans up the now unused `filter_map`. Differential Revision: https://phab.mercurial-scm.org/D7299

rust-status: improve status performance This change does more things in the parallel loop, refactors the file-level logic into two functions for added clarity. This bit of Rust code takes 55ms to execute on a repo where the stat'ing part of Valentin's fast path takes 40ms. While the code differs a bit and it's hard to get an exact measurement of how much of a performance impact it has, I can be fairly certain that this implementation is *at worse* twice as slow. Differential Revision: https://phab.mercurial-scm.org/D7254

bookmarks: accept explicit -r 'wdir()' when adding new bookmarks (issue6218) Even though the bookmark semantics can't be fully encoded to the virtual working changeset idea, the active bookmark can be considered a bookmark of the working revision. Before, 'tgt' was None, and changes=[(bm, None)] means deleting a bookmark named 'bm'.

bookmarks: use changectx instead of remembering hex of hidden revision It should be better to not depend on the ctx variable which was assigned conditionally.

bookmarks: resolve target revision out of the bookmarks loop The target revision doesn't depend on the bookmark to be added.

bookmarks: fix handling of multiple bookmarks with one to be deactivated Before, "hg bookmark --inactive Z Y" would ignore "Y" if "Z" were currently active. I'm pretty sure it is a bug.

py3: avoid iterating over a literal bytes in highlight In Python 3, iterating over a bytes literal yields integers. Since we use the value in `text.replace()`, this fails on Python 3 with the following trackback: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/mercurial/hgweb/hgwebdir_mod.py", line 378, in run_wsgi for r in self._runwsgi(req, res): File "/usr/lib/python3/dist-packages/mercurial/hgweb/hgweb_mod.py", line 326, in run_wsgi for r in self._runwsgi(req, res, repo): File "/usr/lib/python3/dist-packages/mercurial/hgweb/hgweb_mod.py", line 449, in _runwsgi return getattr(webcommands, cmd)(rctx) File "/usr/lib/python3/dist-packages/mercurial/hgweb/webcommands.py", line 211, in file return _filerevision(web, webutil.filectx(web.repo, web.req)) File "/usr/lib/python3/dist-packages/hgext/highlight/__init__.py", line 72, in filerevision_highlight pygmentize(web, b'fileline', fctx, web.tmpl) File "/usr/lib/python3/dist-packages/hgext/highlight/__init__.py", line 58, in pygmentize field, fctx, style, tmpl, guessfilenameonly=filenameonly File "/usr/lib/python3/dist-packages/hgext/highlight/highlight.py", line 62, in pygmentize text = text.replace(c, b'') TypeError: a bytes-like object is required, not 'int'

import-checker: allow 'from typing import ...' Suppresses the following error in test-check-module-imports.t: mercurial/encoding.py:24: relative import of stdlib module

match: drop support for passing '.' for root dir to visit*() methods We said we'd drop support for it after 5.1, so it's time to clean it up now. Differential Revision: https://phab.mercurial-scm.org/D7249

tests: use time.time() for relative start and stop times os.times() does not work on Windows. This was resulting in the test start, stop, and duration times being reported as 0. This commit swaps in time.time() for wall clock measurements. This isn't ideal, as time.time() is not monotonic. But Python 2.7 does not have a monotonic timer that works on Windows. So it is the best we have which is trivially usable. And test times aren't terribly important, so variances due to clock skew are arguably acceptable. Differential Revision: https://phab.mercurial-scm.org/D7126