merge with stable

run-tests: find python binary on Python 3 (issue6361) Return strings from _findprogram as all callers expect unicode strings. Previously the check in _usecorrectpython agains sysexecutable was always false on Python 3. Differential Revision: https://phab.mercurial-scm.org/D8674

tests: ignore possible diagnostics from gpg 2.2 Differential Revision: https://phab.mercurial-scm.org/D8672

Added signature for changeset 0ea9c86fac89

Added tag 5.4.2 for changeset 0ea9c86fac89

convert: handle percent-encoded bytes in file URLs like Subversion 75b59d221aa3 added most of the code that gets removed by this patch. It helped making progress on Python 3, but the reasoning was wrong in many ways. I tried to retract it while it was queued, but it was too late. Back then, I was asssuming that what happened on Python 2 (preserving bytes) is correct and my Python 3 change is a hack. However it turned out that Subversion interprets percent-encoded bytes as UTF-8. Accepting the same format as Subversion is a good idea. Consistency with urlreq.pathname2url() (as described in the removed comment) doesn’t matter because that function is only used for passing paths to urllib. This is not a backwards-incompatible change because before 5c0d5b48e58c, non-ASCII filenames didn’t work at all on Python 2. When the locale encoding is ISO-8859-15, `svn` accepts `file:///tmp/a%E2%82%AC` for `/tmp/a€`. Before this patch, this was the case for this extension on Python 3, but not on Python 2. This patch makes it work like with `svn` on both Python 2 and Python 3.

convert: add docstring on convert.subversion.geturl() The function is unusual for a bytes-handling function in Mercurial because it can’t handle arbitrary bytes. Therefore we should document this fact. Pointed out by Yuya Nishihara while reviewing e3b19004087a.

ui: add option to timestamp status and diagnostic messages Differential Revision: https://phab.mercurial-scm.org/D8640

tests: use path inside test dir This will make the diff for the next patch less noisy.

convert: convert URLs to UTF-8 for Subversion Preamble: for comprehension, note that the `path` of geturl() would better be called `path_or_url` (the argument of the call of getsvn() is called `url`). For HTTP(S) URLs, the changes don’t make a difference, as they are restricted to ASCII. For file URLs, the reasoning is the same as for paths: we have to roundtrip with what Subversion is doing. When the locale encoding is ISO-8859-15, trying to convert a SVN repo `file:///tmp/a€` failed before like this: file:///tmp/a%A4 does not look like a Subversion repository to libsvn version 1.14.0 Decoding the path using the locale encoding can fail. In this case, we have to bail out, as Subversion won’t be able to do anything useful with the path.

convert: correctly convert paths to UTF-8 for Subversion The previous code using encoding.tolocal() only worked by chance in these situations: * The string is ASCII: The fast path was triggered and the string was returned unmodified. * The local encoding is UTF-8: The source and target encoding is the same. * The string is not valid UTF-8 and the native encoding is ISO-8859-1: If the string doesn’t decode using UTF-8, ISO-8859-1 is tried as a fallback. During `hg convert`, the local encoding is always UTF-8. The irony is that in this case, encoding.tolocal() behaves like what someone would expect the reverse function, encoding.fromlocal(), to do. When the locale encoding is ISO-8859-15, trying to convert a SVN repo `/tmp/a€` failed before like this: file:///tmp/a%C2%A4 does not look like a Subversion repository to libsvn version 1.14.0 The correct URL is `file:///tmp/a%E2%82%AC`. Unlike previously (with the ISO-8859-1 fallback), decoding the path using the locale encoding can fail. In this case, we have to bail out, as Subversion won’t be able to do anything useful with the path.

py3: pass URL as str Before the patch, HTTP(S) URLs were never recognized as a Subversion repository on Python 3.

convert: bail out in Subversion source if encountering non-ASCII HTTP(S) URL Before this patch, in the tested case, urllib raised `httplib.InvalidURL: URL can't contain control characters. '/\xff/!svn/ver/0/.svn' (found at least '\xff')`, which resulted in that the URL was never recognized as a Subversion repository. This patch adds a check that bails out if the URL contains non-ASCII characters. The warning is not overly user-friendly, but giving the user something to type into a search engine is definitively better than not explaining why the repository was not recognized. We could support non-ASCII chracters by quoting them before passing them to urllib. However, we would want to be compatible with what the `svn` command does, which converts the URL from the locale encoding to UTF-8, percent-encodes it and sends it to the server. If the locale encoding is not UTF-8, the behavior is IMHO not very intuitive, as the `svn` command may send different (percent-encoded) octets than what was passed on the console. Instead of copying this behavior, we better leave it forbidden.

merge with stable

compat: back out a25343d16ebe (initialize LC_CTYPE locale on all Python ...) As Yuya Nishihara pointed out, setting LC_CTYPE changes the behavior of some str methods on Python 2.

curses: back out d2227d4c9e6b (do not initialize LC_ALL to user settings) The changeset was based on a25343d16ebe, which will be backed out, too. Another fix for the problem will be resubmitted to the stable branch.

test: redirect stderr so warning messages don't change output (issue6237) clone and commit race for the lock, and if commit has to wait more than a second it prints a warning to stderr. Since this is somewhat expected here, silence it. Differential Revision: https://phab.mercurial-scm.org/D8664

locks: expect repo lock, not wlock, when writing to .hg/strip-backup/ There should be no need for a working copy lock when creating (or reading) bundles in `.hg/strip-backup/` since they don't affect the working copy. I noticed this because we have an extension that tries to strip some revisions while holding only a repo lock. I guess we have no such cases in core, which seems a bit surprising. Maybe we always take a wlock at a higher level so the working copy is not updated while the target commit is being stripped. Differential Revision: https://phab.mercurial-scm.org/D8666

graft: leverage cmdutil.check_incompatible_arguments() for --abort/--stop Differential Revision: https://phab.mercurial-scm.org/D8669

run-tests: fix escapes with conditions Before this fix, escapes with conditions in tests failed like this on Python 3: $ $PYTHON -c 'from mercurial.utils.procutil import stdout; stdout.write(b"\xff")' - \xff (no-eol) (esc) (true !) + \xff (no-eol) (esc) The unicode_escape encoding decodes br'\xff' to u'\xff'. To convert the first 256 code points to bytes with the same ordinal, the latin-1 encoding must be used. Escapes without conditions already worked before on Python 3, but not through `el == l` a few lines below the changed line in run-tests.py. I didn’t investigate further.

convert: set LC_CTYPE around calls to Subversion bindings The Subversion bindings require that LC_CTYPE is set. However, we don’t want to set it all the time, as it changes the behavior of str methods on Python 2. The taken approach is hopefully fine-grained enough to not trigger any locale-specfic behavior of the str methods and coarse-grained enough to not clutter the code. Emulating the with-statement behavior in before() and after() should be safe, as after() is always called when before() is called. hgext.convert.hg takes a similar approach.

curses: do not initialize LC_ALL to user settings (issue6358) 701341f57ceb moved the setlocale() call to right before curses was used. This didn’t fully solve the problem it was supposed to solve (locale-dependent functions, like date formatting/parsing and str methods on Python 2), but only postponed it. Initializing LC_CTYPE seems to be sufficient for curses to work correctly. Therefore LC_CTYPE is set while curses is used and reset afterwards. Some locale-dependent str methods might behave differently on Python 2 while curses is used, but that shouldn’d be a problem.

graft: leverage cmdutil.check_incompatible_arguments() for --no-commit Differential Revision: https://phab.mercurial-scm.org/D8668

graft: leverage cmdutil.check_at_most_one_arg() for --abort/--stop/--continue Differential Revision: https://phab.mercurial-scm.org/D8667

version: sort extensions by name in verbose mode External extensions can be assigned any name, but presumably most enabled extensions will be internal ones and having them sorted makes it easier to find specific ones if the list is long. The lists in `hg help extensions` are already sorted. Differential Revision: https://phab.mercurial-scm.org/D8671

crecord: stop trying to import wcurses The original import of crecord in 2008 already said "I have no idea if wcurses works with crecord...". The last reference to a Python package called wcurses is https://web.archive.org/web/20101025073658/http://adamv.com/dev/python/curses/. However, the Python package from there is called "curses" and not "wcurses". I didn’t find any evidence that it ever worked.

debian: support building a single deb for multiple py3 versions Around transitions from one python minor version to another (such as 3.7 to 3.8), the current packaging can be slightly problematic - it produces a `control` file that requires that the version of `python3` that's installed be exactly the one that was used on the build machine for the `mercurial` package, by containing a line like: Depends: sensible-utils, libc6 (>= 2.14), python3 (<< 3.8), python3 (>= 3.7~), python3:any (>= 3.5~) This is because it "knows" we only built for v3.7, which is the current default on my system. By building the native components for multiple versions, we can make it produce a line like this, which is compatible with 3.7 AND 3.8: Depends: sensible-utils, libc6 (>= 2.14), python3 (<< 3.9), python3 (>= 3.7~), python3:any (>= 3.5~) This isn't *normally* required, so I'm not making it the default. For those that receive their python3 and mercurial packages from their distro, and/or don't have to worry about a situation where the team that manages the python3 installation isn't the same as the team that manages the mercurial installation, this is probably not necessary. I chose the names `DEB_HG_*` because `DEB_*` is passed through `debuild` automatically (otherwise we'd have to explicitly allow the options through, which is a nuisance), and the `HG` part is to make it clear that this isn't a "standard" debian option that other packages might respect. Test Plan: 1. "nothing changed": - built a deb without these changes - built a deb with these changes but everything at the default - used diffoscope to compare, all differences were due to timestamps 2. "explicit is the same as implicit" (single version) - built a deb with everything at the default - built a deb with DEB_HG_PYTHON_VERSIONS=3.7 - used diffoscope to compare, all differences were due to timestamps 3. "explicit is the same as implicit" (multi version) - built a deb with DEB_HG_MULTI_VERSION=1 - built a deb with DEB_HG_PYTHON_VERSIONS=3.7 - used diffoscope to compare, all differences were due to timestamps 4. (single version, 3.7) doesn't work with python3.8 - `/usr/bin/python3.7 /usr/bin/hg debuginstall` works - `/usr/bin/python3.8 /usr/bin/hg debuginstall` crashes 5. (multi version, 3.7 + 3.8) - `/usr/bin/python3.7 /usr/bin/hg debuginstall` works - `/usr/bin/python3.8 /usr/bin/hg debuginstall` works Differential Revision: https://phab.mercurial-scm.org/D8642

merge with stable

hgweb: encode WSGI environment like OS environment Previously, the WSGI environment keys and values were encoded using latin-1. This resulted in a crash if a WSGI environment key or value could not be encoded using latin-1. On Unix, the OS environment is byte-based. Therefore we should do the reverse of what Python does for os.environ. On Windows, there’s no native byte-based OS environment. Therefore we should do the same as what mercurial.encoding does with the OS environment.

hgweb: deduplicate code A following patch will change the way keys and values are encoded. To reduce the diff, I’ve split off the uninteresting part.