Manuel Jacob <me@manueljacob.de> [Tue, 30 Jun 2020 07:23:29 +0200] rev 44835
convert: handle percent-encoded bytes in file URLs like Subversion
75b59d221aa3 added most of the code that gets removed by this patch. It helped
making progress on Python 3, but the reasoning was wrong in many ways. I tried
to retract it while it was queued, but it was too late.
Back then, I was asssuming that what happened on Python 2 (preserving bytes) is
correct and my Python 3 change is a hack. However it turned out that Subversion
interprets percent-encoded bytes as UTF-8. Accepting the same format as
Subversion is a good idea.
Consistency with urlreq.pathname2url() (as described in the removed comment)
doesn’t matter because that function is only used for passing paths to urllib.
This is not a backwards-incompatible change because before 5c0d5b48e58c,
non-ASCII filenames didn’t work at all on Python 2.
When the locale encoding is ISO-8859-15, `svn` accepts `file:///tmp/a%E2%82%AC`
for `/tmp/a€`. Before this patch, this was the case for this extension on
Python 3, but not on Python 2. This patch makes it work like with `svn` on both
Python 2 and Python 3.
Manuel Jacob <me@manueljacob.de> [Tue, 30 Jun 2020 16:39:45 +0200] rev 44834
convert: add docstring on convert.subversion.geturl()
The function is unusual for a bytes-handling function in Mercurial because it
can’t handle arbitrary bytes. Therefore we should document this fact.
Pointed out by Yuya Nishihara while reviewing e3b19004087a.
Manuel Jacob <me@manueljacob.de> [Tue, 30 Jun 2020 01:32:17 +0200] rev 44833
tests: use path inside test dir
This will make the diff for the next patch less noisy.
Manuel Jacob <me@manueljacob.de> [Tue, 30 Jun 2020 05:30:47 +0200] rev 44832
convert: convert URLs to UTF-8 for Subversion
Preamble: for comprehension, note that the `path` of geturl() would better be
called `path_or_url` (the argument of the call of getsvn() is called `url`).
For HTTP(S) URLs, the changes don’t make a difference, as they are restricted to
ASCII.
For file URLs, the reasoning is the same as for paths: we have to roundtrip with
what Subversion is doing.
When the locale encoding is ISO-8859-15, trying to convert a SVN repo
`file:///tmp/a€` failed before like this:
file:///tmp/a%A4 does not look like a Subversion repository to libsvn version 1.14.0
Decoding the path using the locale encoding can fail. In this case, we have to
bail out, as Subversion won’t be able to do anything useful with the path.
Manuel Jacob <me@manueljacob.de> [Mon, 29 Jun 2020 15:03:36 +0200] rev 44831
convert: correctly convert paths to UTF-8 for Subversion
The previous code using encoding.tolocal() only worked by chance in these
situations:
* The string is ASCII: The fast path was triggered and the string was returned
unmodified.
* The local encoding is UTF-8: The source and target encoding is the same.
* The string is not valid UTF-8 and the native encoding is ISO-8859-1: If the
string doesn’t decode using UTF-8, ISO-8859-1 is tried as a fallback. During
`hg convert`, the local encoding is always UTF-8. The irony is that in this
case, encoding.tolocal() behaves like what someone would expect the reverse
function, encoding.fromlocal(), to do.
When the locale encoding is ISO-8859-15, trying to convert a SVN repo `/tmp/a€`
failed before like this:
file:///tmp/a%C2%A4 does not look like a Subversion repository to libsvn version 1.14.0
The correct URL is `file:///tmp/a%E2%82%AC`.
Unlike previously (with the ISO-8859-1 fallback), decoding the path using the
locale encoding can fail. In this case, we have to bail out, as Subversion
won’t be able to do anything useful with the path.
Manuel Jacob <me@manueljacob.de> [Tue, 30 Jun 2020 05:04:36 +0200] rev 44830
py3: pass URL as str
Before the patch, HTTP(S) URLs were never recognized as a Subversion repository
on Python 3.
Manuel Jacob <me@manueljacob.de> [Tue, 30 Jun 2020 04:55:52 +0200] rev 44829
convert: bail out in Subversion source if encountering non-ASCII HTTP(S) URL
Before this patch, in the tested case, urllib raised `httplib.InvalidURL: URL
can't contain control characters. '/\xff/!svn/ver/0/.svn' (found at least
'\xff')`, which resulted in that the URL was never recognized as a Subversion
repository.
This patch adds a check that bails out if the URL contains non-ASCII characters.
The warning is not overly user-friendly, but giving the user something to type
into a search engine is definitively better than not explaining why the
repository was not recognized.
We could support non-ASCII chracters by quoting them before passing them to
urllib. However, we would want to be compatible with what the `svn` command
does, which converts the URL from the locale encoding to UTF-8, percent-encodes
it and sends it to the server. If the locale encoding is not UTF-8, the
behavior is IMHO not very intuitive, as the `svn` command may send different
(percent-encoded) octets than what was passed on the console. Instead of
copying this behavior, we better leave it forbidden.
Manuel Jacob <me@manueljacob.de> [Mon, 29 Jun 2020 02:05:12 +0200] rev 44828
run-tests: fix escapes with conditions
Before this fix, escapes with conditions in tests failed like this on Python 3:
$ $PYTHON -c 'from mercurial.utils.procutil import stdout; stdout.write(b"\xff")'
- \xff (no-eol) (esc) (true !)
+ \xff (no-eol) (esc)
The unicode_escape encoding decodes br'\xff' to u'\xff'. To convert the first
256 code points to bytes with the same ordinal, the latin-1 encoding must be
used.
Escapes without conditions already worked before on Python 3, but not through
`el == l` a few lines below the changed line in run-tests.py. I didn’t
investigate further.
Manuel Jacob <me@manueljacob.de> [Sun, 28 Jun 2020 18:02:45 +0200] rev 44827
convert: set LC_CTYPE around calls to Subversion bindings
The Subversion bindings require that LC_CTYPE is set. However, we don’t want to
set it all the time, as it changes the behavior of str methods on Python 2. The
taken approach is hopefully fine-grained enough to not trigger any
locale-specfic behavior of the str methods and coarse-grained enough to not
clutter the code.
Emulating the with-statement behavior in before() and after() should be safe, as
after() is always called when before() is called. hgext.convert.hg takes a
similar approach.
Manuel Jacob <me@manueljacob.de> [Sun, 28 Jun 2020 18:02:45 +0200] rev 44826
curses: do not initialize LC_ALL to user settings (issue6358)
701341f57ceb moved the setlocale() call to right before curses was used. This
didn’t fully solve the problem it was supposed to solve (locale-dependent
functions, like date formatting/parsing and str methods on Python 2), but only
postponed it.
Initializing LC_CTYPE seems to be sufficient for curses to work correctly.
Therefore LC_CTYPE is set while curses is used and reset afterwards. Some
locale-dependent str methods might behave differently on Python 2 while curses
is used, but that shouldn’d be a problem.
Manuel Jacob <me@manueljacob.de> [Thu, 25 Jun 2020 03:46:07 +0200] rev 44825
hgweb: encode WSGI environment like OS environment
Previously, the WSGI environment keys and values were encoded using latin-1.
This resulted in a crash if a WSGI environment key or value could not be encoded
using latin-1.
On Unix, the OS environment is byte-based. Therefore we should do the reverse of
what Python does for os.environ.
On Windows, there’s no native byte-based OS environment. Therefore we should do
the same as what mercurial.encoding does with the OS environment.
Manuel Jacob <me@manueljacob.de> [Thu, 25 Jun 2020 03:10:13 +0200] rev 44824
hgweb: deduplicate code
A following patch will change the way keys and values are encoded. To reduce the
diff, I’ve split off the uninteresting part.
Joerg Sonnenberger <joerg@bec.de> [Tue, 23 Jun 2020 16:07:18 +0200] rev 44823
share: provide a more useful text for hg help
Differential Revision: https://phab.mercurial-scm.org/D8651
Martin von Zweigbergk <martinvonz@google.com> [Thu, 25 Jun 2020 00:06:23 -0700] rev 44822
procutil: make recent fix for zombies compatible with py2
The fix in ed684a82e29b (procutil: always waiting on child processes
to prevent zombies with 'hg serve', 2020-05-07) works only on Python 3
because it passes a `daemon` argument to `threading.Thread()`. Python
2 requires you to assign to the `.daemon` property instead. Python 3
also seems to support that, so this patch fixes the code by
unconditionally using the old form.
Differential Revision: https://phab.mercurial-scm.org/D8657