rust-chg: upgrade to futures-0.3 based libraries And do some trivial fixes: - BytesMut::put_u32_be() -> put_u32() - tokio_process -> tokio::process, CommandExt -> Command, spawn_async() -> spawn(), stdin() -> stdin - tokio_timer::sleep() -> tokio::time::delay_for() Differential Revision: https://phab.mercurial-scm.org/D8441

rust-chg: exclude futures-dependent modules from build and break things It's impractical to upgrade the codebase incrementally since futures 0.1 and 0.3 APIs are fundamentally different. So this patch temporarily excludes futures-dependent modules from the build. These modules will be upgraded and re-enabled one by one. Differential Revision: https://phab.mercurial-scm.org/D8440

commit: tell user what to do with .hg/last-message.txt I have always assumed that the message will be reused by the next `hg commit`, but it seems it's just silently dropped on the next commit. Let's try to be more helpful by telling the user that they have to manually tell hg to reuse it. The file will still be lost if the user runs some other operation in between (like a non-in-memory rebase). That will be fixed once we've switched all operations to be in-memory :) I didn't include `$(hg root)/` in the path in the message to the user because that would have made the message too long. Hopefully the user will figure that part out themselves. Differential Revision: https://phab.mercurial-scm.org/D8463

test-check-rust-format: specify --edition=2018 rustfmt doesn't read Cargo.toml unless it's executed by cargo. https://github.com/rust-lang/rustfmt#rusts-editions

convert: use bytes for value in extra dict The keys and values of the extra dict are expected to be bytes. Before this fix, there was a crash in mercurial.changelog.encodeextra().

archival: abort if compression method is unavailable `tarfile.CompressionError` is documented to be the "exception for unavailable compression methods". Also, make tests conditional on whether the lzma module is available or not.

demandimport: ignore `lzma` module for demandimport This makes importing the module fail if the `_lzma` module is not present. This makes e.g. tarfile correctly recognize if LZMA support is not present. It changes the exception File "/usr/lib/python3.6/tarfile.py", line 1694, in xzopen fileobj = lzma.LZMAFile(fileobj or name, mode, preset=preset) AttributeError: module 'lzma' has no attribute 'LZMAFile' to the more correct exception File "/usr/lib/python3.6/tarfile.py", line 1692, in xzopen raise CompressionError("lzma module is not available") tarfile.CompressionError: lzma module is not available Also, it prevents that the error "abort: No module named '_lzma'!" is shown when a development warning is to be shown. The reason why that happened is that for showing the warning, we get information about the stack frames from the inspect module, which accesses the `__file__` attribute of all modules in `sys.modules` to build some cache, causing all modules (including `lzma`) to be imported.

py3: fix crash when server address is 0.0.0.0 (issue6362) `socket.getfqdn()` assumes that the name is passed as `str` on Python 3 and always returns `str` in this case. Mercurial passed `bytes` (but still expected a `str` result), which worked by chance in many cases, except for e.g. b'0.0.0.0', which was returned unchanged, breaking later code. Instead of calling `socket.getfqdn()`, we can also use `self.server_name` from the base `HTTPServer` class, which already stores the FQDN of the locally-bound socket name (see `BaseHTTPServer.py` in the Python 2 stdlib and `http/server.py` in the Python 3 stdlib).

run-tests: find python binary on Python 3 (issue6361) Return strings from _findprogram as all callers expect unicode strings. Previously the check in _usecorrectpython agains sysexecutable was always false on Python 3. Differential Revision: https://phab.mercurial-scm.org/D8674

tests: ignore possible diagnostics from gpg 2.2 Differential Revision: https://phab.mercurial-scm.org/D8672

Added signature for changeset 0ea9c86fac89

Added tag 5.4.2 for changeset 0ea9c86fac89

convert: handle percent-encoded bytes in file URLs like Subversion 75b59d221aa3 added most of the code that gets removed by this patch. It helped making progress on Python 3, but the reasoning was wrong in many ways. I tried to retract it while it was queued, but it was too late. Back then, I was asssuming that what happened on Python 2 (preserving bytes) is correct and my Python 3 change is a hack. However it turned out that Subversion interprets percent-encoded bytes as UTF-8. Accepting the same format as Subversion is a good idea. Consistency with urlreq.pathname2url() (as described in the removed comment) doesn’t matter because that function is only used for passing paths to urllib. This is not a backwards-incompatible change because before 5c0d5b48e58c, non-ASCII filenames didn’t work at all on Python 2. When the locale encoding is ISO-8859-15, `svn` accepts `file:///tmp/a%E2%82%AC` for `/tmp/a€`. Before this patch, this was the case for this extension on Python 3, but not on Python 2. This patch makes it work like with `svn` on both Python 2 and Python 3.

convert: add docstring on convert.subversion.geturl() The function is unusual for a bytes-handling function in Mercurial because it can’t handle arbitrary bytes. Therefore we should document this fact. Pointed out by Yuya Nishihara while reviewing e3b19004087a.

tests: use path inside test dir This will make the diff for the next patch less noisy.

convert: convert URLs to UTF-8 for Subversion Preamble: for comprehension, note that the `path` of geturl() would better be called `path_or_url` (the argument of the call of getsvn() is called `url`). For HTTP(S) URLs, the changes don’t make a difference, as they are restricted to ASCII. For file URLs, the reasoning is the same as for paths: we have to roundtrip with what Subversion is doing. When the locale encoding is ISO-8859-15, trying to convert a SVN repo `file:///tmp/a€` failed before like this: file:///tmp/a%A4 does not look like a Subversion repository to libsvn version 1.14.0 Decoding the path using the locale encoding can fail. In this case, we have to bail out, as Subversion won’t be able to do anything useful with the path.

convert: correctly convert paths to UTF-8 for Subversion The previous code using encoding.tolocal() only worked by chance in these situations: * The string is ASCII: The fast path was triggered and the string was returned unmodified. * The local encoding is UTF-8: The source and target encoding is the same. * The string is not valid UTF-8 and the native encoding is ISO-8859-1: If the string doesn’t decode using UTF-8, ISO-8859-1 is tried as a fallback. During `hg convert`, the local encoding is always UTF-8. The irony is that in this case, encoding.tolocal() behaves like what someone would expect the reverse function, encoding.fromlocal(), to do. When the locale encoding is ISO-8859-15, trying to convert a SVN repo `/tmp/a€` failed before like this: file:///tmp/a%C2%A4 does not look like a Subversion repository to libsvn version 1.14.0 The correct URL is `file:///tmp/a%E2%82%AC`. Unlike previously (with the ISO-8859-1 fallback), decoding the path using the locale encoding can fail. In this case, we have to bail out, as Subversion won’t be able to do anything useful with the path.

py3: pass URL as str Before the patch, HTTP(S) URLs were never recognized as a Subversion repository on Python 3.

convert: bail out in Subversion source if encountering non-ASCII HTTP(S) URL Before this patch, in the tested case, urllib raised `httplib.InvalidURL: URL can't contain control characters. '/\xff/!svn/ver/0/.svn' (found at least '\xff')`, which resulted in that the URL was never recognized as a Subversion repository. This patch adds a check that bails out if the URL contains non-ASCII characters. The warning is not overly user-friendly, but giving the user something to type into a search engine is definitively better than not explaining why the repository was not recognized. We could support non-ASCII chracters by quoting them before passing them to urllib. However, we would want to be compatible with what the `svn` command does, which converts the URL from the locale encoding to UTF-8, percent-encodes it and sends it to the server. If the locale encoding is not UTF-8, the behavior is IMHO not very intuitive, as the `svn` command may send different (percent-encoded) octets than what was passed on the console. Instead of copying this behavior, we better leave it forbidden.

run-tests: fix escapes with conditions Before this fix, escapes with conditions in tests failed like this on Python 3: $ $PYTHON -c 'from mercurial.utils.procutil import stdout; stdout.write(b"\xff")' - \xff (no-eol) (esc) (true !) + \xff (no-eol) (esc) The unicode_escape encoding decodes br'\xff' to u'\xff'. To convert the first 256 code points to bytes with the same ordinal, the latin-1 encoding must be used. Escapes without conditions already worked before on Python 3, but not through `el == l` a few lines below the changed line in run-tests.py. I didn’t investigate further.

convert: set LC_CTYPE around calls to Subversion bindings The Subversion bindings require that LC_CTYPE is set. However, we don’t want to set it all the time, as it changes the behavior of str methods on Python 2. The taken approach is hopefully fine-grained enough to not trigger any locale-specfic behavior of the str methods and coarse-grained enough to not clutter the code. Emulating the with-statement behavior in before() and after() should be safe, as after() is always called when before() is called. hgext.convert.hg takes a similar approach.

curses: do not initialize LC_ALL to user settings (issue6358) 701341f57ceb moved the setlocale() call to right before curses was used. This didn’t fully solve the problem it was supposed to solve (locale-dependent functions, like date formatting/parsing and str methods on Python 2), but only postponed it. Initializing LC_CTYPE seems to be sufficient for curses to work correctly. Therefore LC_CTYPE is set while curses is used and reset afterwards. Some locale-dependent str methods might behave differently on Python 2 while curses is used, but that shouldn’d be a problem.

hgweb: encode WSGI environment like OS environment Previously, the WSGI environment keys and values were encoded using latin-1. This resulted in a crash if a WSGI environment key or value could not be encoded using latin-1. On Unix, the OS environment is byte-based. Therefore we should do the reverse of what Python does for os.environ. On Windows, there’s no native byte-based OS environment. Therefore we should do the same as what mercurial.encoding does with the OS environment.

hgweb: deduplicate code A following patch will change the way keys and values are encoded. To reduce the diff, I’ve split off the uninteresting part.