chg: suppress OSError in _restoreio() and add some logging (issue6330) According to issue6330, running chg on heavy loaded systems can lead to following error: ``` Traceback (most recent call last): File "path-to-hg/mercurial/commandserver.py", line 650, in _acceptnewconnection self._runworker(conn) File "path-to-hg/mercurial/commandserver.py", line 701, in _runworker prereposetups=[self._reposetup], File "path-to-hg/mercurial/commandserver.py", line 470, in _serverequest sv.cleanup() File "path-to-hg/mercurial/chgserver.py", line 381, in cleanup self._restoreio() File "path-to-hg/mercurial/chgserver.py", line 444, in _restoreio os.dup2(fd, fp.fileno()) OSError: [Errno 16] Device or resource busy ``` [man dup2] indicates that, on Linux, EBUSY comes from a race condition between open() and dup2(). However it's not clear why open() race occurred for newfd=stdin/out/err. We suppress the OSError in _restoreio() since the forked worker process will finish anyway and add some logging. Thanks to Mitchell Plamann for a detailed bug description and Yuya Nishihara for suggesting the fix.

py3: fix crash when server address is 0.0.0.0 (issue6362) `socket.getfqdn()` assumes that the name is passed as `str` on Python 3 and always returns `str` in this case. Mercurial passed `bytes` (but still expected a `str` result), which worked by chance in many cases, except for e.g. b'0.0.0.0', which was returned unchanged, breaking later code. Instead of calling `socket.getfqdn()`, we can also use `self.server_name` from the base `HTTPServer` class, which already stores the FQDN of the locally-bound socket name (see `BaseHTTPServer.py` in the Python 2 stdlib and `http/server.py` in the Python 3 stdlib).

ui: fix Python 2.7 support for ui.timestamp-output Differential Revision: https://phab.mercurial-scm.org/D8675

merge with stable

run-tests: find python binary on Python 3 (issue6361) Return strings from _findprogram as all callers expect unicode strings. Previously the check in _usecorrectpython agains sysexecutable was always false on Python 3. Differential Revision: https://phab.mercurial-scm.org/D8674

tests: ignore possible diagnostics from gpg 2.2 Differential Revision: https://phab.mercurial-scm.org/D8672

Added signature for changeset 0ea9c86fac89

Added tag 5.4.2 for changeset 0ea9c86fac89

convert: handle percent-encoded bytes in file URLs like Subversion 75b59d221aa3 added most of the code that gets removed by this patch. It helped making progress on Python 3, but the reasoning was wrong in many ways. I tried to retract it while it was queued, but it was too late. Back then, I was asssuming that what happened on Python 2 (preserving bytes) is correct and my Python 3 change is a hack. However it turned out that Subversion interprets percent-encoded bytes as UTF-8. Accepting the same format as Subversion is a good idea. Consistency with urlreq.pathname2url() (as described in the removed comment) doesn’t matter because that function is only used for passing paths to urllib. This is not a backwards-incompatible change because before 5c0d5b48e58c, non-ASCII filenames didn’t work at all on Python 2. When the locale encoding is ISO-8859-15, `svn` accepts `file:///tmp/a%E2%82%AC` for `/tmp/a€`. Before this patch, this was the case for this extension on Python 3, but not on Python 2. This patch makes it work like with `svn` on both Python 2 and Python 3.

convert: add docstring on convert.subversion.geturl() The function is unusual for a bytes-handling function in Mercurial because it can’t handle arbitrary bytes. Therefore we should document this fact. Pointed out by Yuya Nishihara while reviewing e3b19004087a.

ui: add option to timestamp status and diagnostic messages Differential Revision: https://phab.mercurial-scm.org/D8640

tests: use path inside test dir This will make the diff for the next patch less noisy.

convert: convert URLs to UTF-8 for Subversion Preamble: for comprehension, note that the `path` of geturl() would better be called `path_or_url` (the argument of the call of getsvn() is called `url`). For HTTP(S) URLs, the changes don’t make a difference, as they are restricted to ASCII. For file URLs, the reasoning is the same as for paths: we have to roundtrip with what Subversion is doing. When the locale encoding is ISO-8859-15, trying to convert a SVN repo `file:///tmp/a€` failed before like this: file:///tmp/a%A4 does not look like a Subversion repository to libsvn version 1.14.0 Decoding the path using the locale encoding can fail. In this case, we have to bail out, as Subversion won’t be able to do anything useful with the path.

convert: correctly convert paths to UTF-8 for Subversion The previous code using encoding.tolocal() only worked by chance in these situations: * The string is ASCII: The fast path was triggered and the string was returned unmodified. * The local encoding is UTF-8: The source and target encoding is the same. * The string is not valid UTF-8 and the native encoding is ISO-8859-1: If the string doesn’t decode using UTF-8, ISO-8859-1 is tried as a fallback. During `hg convert`, the local encoding is always UTF-8. The irony is that in this case, encoding.tolocal() behaves like what someone would expect the reverse function, encoding.fromlocal(), to do. When the locale encoding is ISO-8859-15, trying to convert a SVN repo `/tmp/a€` failed before like this: file:///tmp/a%C2%A4 does not look like a Subversion repository to libsvn version 1.14.0 The correct URL is `file:///tmp/a%E2%82%AC`. Unlike previously (with the ISO-8859-1 fallback), decoding the path using the locale encoding can fail. In this case, we have to bail out, as Subversion won’t be able to do anything useful with the path.

py3: pass URL as str Before the patch, HTTP(S) URLs were never recognized as a Subversion repository on Python 3.

convert: bail out in Subversion source if encountering non-ASCII HTTP(S) URL Before this patch, in the tested case, urllib raised `httplib.InvalidURL: URL can't contain control characters. '/\xff/!svn/ver/0/.svn' (found at least '\xff')`, which resulted in that the URL was never recognized as a Subversion repository. This patch adds a check that bails out if the URL contains non-ASCII characters. The warning is not overly user-friendly, but giving the user something to type into a search engine is definitively better than not explaining why the repository was not recognized. We could support non-ASCII chracters by quoting them before passing them to urllib. However, we would want to be compatible with what the `svn` command does, which converts the URL from the locale encoding to UTF-8, percent-encodes it and sends it to the server. If the locale encoding is not UTF-8, the behavior is IMHO not very intuitive, as the `svn` command may send different (percent-encoded) octets than what was passed on the console. Instead of copying this behavior, we better leave it forbidden.

merge with stable

compat: back out a25343d16ebe (initialize LC_CTYPE locale on all Python ...) As Yuya Nishihara pointed out, setting LC_CTYPE changes the behavior of some str methods on Python 2.

curses: back out d2227d4c9e6b (do not initialize LC_ALL to user settings) The changeset was based on a25343d16ebe, which will be backed out, too. Another fix for the problem will be resubmitted to the stable branch.

test: redirect stderr so warning messages don't change output (issue6237) clone and commit race for the lock, and if commit has to wait more than a second it prints a warning to stderr. Since this is somewhat expected here, silence it. Differential Revision: https://phab.mercurial-scm.org/D8664

locks: expect repo lock, not wlock, when writing to .hg/strip-backup/ There should be no need for a working copy lock when creating (or reading) bundles in `.hg/strip-backup/` since they don't affect the working copy. I noticed this because we have an extension that tries to strip some revisions while holding only a repo lock. I guess we have no such cases in core, which seems a bit surprising. Maybe we always take a wlock at a higher level so the working copy is not updated while the target commit is being stripped. Differential Revision: https://phab.mercurial-scm.org/D8666

graft: leverage cmdutil.check_incompatible_arguments() for --abort/--stop Differential Revision: https://phab.mercurial-scm.org/D8669

run-tests: fix escapes with conditions Before this fix, escapes with conditions in tests failed like this on Python 3: $ $PYTHON -c 'from mercurial.utils.procutil import stdout; stdout.write(b"\xff")' - \xff (no-eol) (esc) (true !) + \xff (no-eol) (esc) The unicode_escape encoding decodes br'\xff' to u'\xff'. To convert the first 256 code points to bytes with the same ordinal, the latin-1 encoding must be used. Escapes without conditions already worked before on Python 3, but not through `el == l` a few lines below the changed line in run-tests.py. I didn’t investigate further.

convert: set LC_CTYPE around calls to Subversion bindings The Subversion bindings require that LC_CTYPE is set. However, we don’t want to set it all the time, as it changes the behavior of str methods on Python 2. The taken approach is hopefully fine-grained enough to not trigger any locale-specfic behavior of the str methods and coarse-grained enough to not clutter the code. Emulating the with-statement behavior in before() and after() should be safe, as after() is always called when before() is called. hgext.convert.hg takes a similar approach.

curses: do not initialize LC_ALL to user settings (issue6358) 701341f57ceb moved the setlocale() call to right before curses was used. This didn’t fully solve the problem it was supposed to solve (locale-dependent functions, like date formatting/parsing and str methods on Python 2), but only postponed it. Initializing LC_CTYPE seems to be sufficient for curses to work correctly. Therefore LC_CTYPE is set while curses is used and reset afterwards. Some locale-dependent str methods might behave differently on Python 2 while curses is used, but that shouldn’d be a problem.

graft: leverage cmdutil.check_incompatible_arguments() for --no-commit Differential Revision: https://phab.mercurial-scm.org/D8668

graft: leverage cmdutil.check_at_most_one_arg() for --abort/--stop/--continue Differential Revision: https://phab.mercurial-scm.org/D8667

version: sort extensions by name in verbose mode External extensions can be assigned any name, but presumably most enabled extensions will be internal ones and having them sorted makes it easier to find specific ones if the list is long. The lists in `hg help extensions` are already sorted. Differential Revision: https://phab.mercurial-scm.org/D8671

crecord: stop trying to import wcurses The original import of crecord in 2008 already said "I have no idea if wcurses works with crecord...". The last reference to a Python package called wcurses is https://web.archive.org/web/20101025073658/http://adamv.com/dev/python/curses/. However, the Python package from there is called "curses" and not "wcurses". I didn’t find any evidence that it ever worked.

debian: support building a single deb for multiple py3 versions Around transitions from one python minor version to another (such as 3.7 to 3.8), the current packaging can be slightly problematic - it produces a `control` file that requires that the version of `python3` that's installed be exactly the one that was used on the build machine for the `mercurial` package, by containing a line like: Depends: sensible-utils, libc6 (>= 2.14), python3 (<< 3.8), python3 (>= 3.7~), python3:any (>= 3.5~) This is because it "knows" we only built for v3.7, which is the current default on my system. By building the native components for multiple versions, we can make it produce a line like this, which is compatible with 3.7 AND 3.8: Depends: sensible-utils, libc6 (>= 2.14), python3 (<< 3.9), python3 (>= 3.7~), python3:any (>= 3.5~) This isn't *normally* required, so I'm not making it the default. For those that receive their python3 and mercurial packages from their distro, and/or don't have to worry about a situation where the team that manages the python3 installation isn't the same as the team that manages the mercurial installation, this is probably not necessary. I chose the names `DEB_HG_*` because `DEB_*` is passed through `debuild` automatically (otherwise we'd have to explicitly allow the options through, which is a nuisance), and the `HG` part is to make it clear that this isn't a "standard" debian option that other packages might respect. Test Plan: 1. "nothing changed": - built a deb without these changes - built a deb with these changes but everything at the default - used diffoscope to compare, all differences were due to timestamps 2. "explicit is the same as implicit" (single version) - built a deb with everything at the default - built a deb with DEB_HG_PYTHON_VERSIONS=3.7 - used diffoscope to compare, all differences were due to timestamps 3. "explicit is the same as implicit" (multi version) - built a deb with DEB_HG_MULTI_VERSION=1 - built a deb with DEB_HG_PYTHON_VERSIONS=3.7 - used diffoscope to compare, all differences were due to timestamps 4. (single version, 3.7) doesn't work with python3.8 - `/usr/bin/python3.7 /usr/bin/hg debuginstall` works - `/usr/bin/python3.8 /usr/bin/hg debuginstall` crashes 5. (multi version, 3.7 + 3.8) - `/usr/bin/python3.7 /usr/bin/hg debuginstall` works - `/usr/bin/python3.8 /usr/bin/hg debuginstall` works Differential Revision: https://phab.mercurial-scm.org/D8642

merge with stable

hgweb: encode WSGI environment like OS environment Previously, the WSGI environment keys and values were encoded using latin-1. This resulted in a crash if a WSGI environment key or value could not be encoded using latin-1. On Unix, the OS environment is byte-based. Therefore we should do the reverse of what Python does for os.environ. On Windows, there’s no native byte-based OS environment. Therefore we should do the same as what mercurial.encoding does with the OS environment.

hgweb: deduplicate code A following patch will change the way keys and values are encoded. To reduce the diff, I’ve split off the uninteresting part.

curses: do not initialize LC_ALL to user settings (issue6358) 701341f57ceb moved the setlocale() call to right before curses was used. This didn’t fully solve the problem it was supposed to solve (locale-dependent functions, like date formatting/parsing), but only postponed it. Initializing LC_CTYPE seems to be sufficient for curses to work correctly. Luckily this is already done at interpreter startup on modern Python versions and, since recently, by Mercurial in the pycompat module in all other cases.

compat: initialize LC_CTYPE locale on all Python versions and platforms Previously, the LC_CTYPE locale was not initialized according to user settings on all Python versions (e.g. never on Python 2) and platforms (e.g. not on some Python < 3.8 on Windows). This broke e.g. non-ASCII filenames passed to the Subversion bindings on Python 2, resulting in error messages like "file:///tmp/a%C3%A4 does not look like a Subversion repository to libsvn version 1.14.0". The following command could be used to test this functionality. Adding it to the test suite would be pointless, as the locale is always set to "C" during test runs. @command(b'check_initial_codeset', norepo=True) def check_initial_codeset(ui): codeset1 = locale.nl_langinfo(locale.CODESET) locale.setlocale(locale.LC_ALL, '') codeset2 = locale.nl_langinfo(locale.CODESET) assert codeset1 == codeset2

merge with stable

merge: don't grab wlock when merging in memory I noticed this because we have an internal extension that does an in-memory rebase while holding only a repo lock, which resulted in a developer warning about the working copy lock being taken after the repo lock. Differential Revision: https://phab.mercurial-scm.org/D8665

pycompat: use os.fsencode() to re-encode sys.argv Historically, the previous code made sense, as Py_EncodeLocale() and fs.fsencode() could possibly use different encodings. However, this is not the case anymore for Python 3.2, which uses the locale encoding as the filesystem encoding (this is not true for later Python versions, but see below). See https://vstinner.github.io/painful-history-python-filesystem-encoding.html for a source and more background information. Using os.fsencode() is safer, as the documentation for sys.argv says that it can be used to get the original bytes. When doing further changes, the Python developers will take care that this continues to work. One concrete case where os.fsencode() is more correct is when enabling Python's UTF-8 mode. Py_DecodeLocale() will use UTF-8 in this case. Our previous code would have encoded it using the locale encoding (which might be different), whereas os.fsencode() will encode it with UTF-8. Since we don’t claim to support the UTF-8 mode, this is not really a bug and the patch can go to the default branch. It might be a good idea to not commit this to the stable branch, as it could in theory introduce regressions.

merge with stable

perf: make `hg perfwrite` more flexible The more flexible command was used recently while finding a solution for a buffering bug (eventually fixed in f9734b2d59cc (the changeset description uses a different benchmark)). In comparison to the previous version, the new version is much more flexible. While using it, the focus was on testing small writes. For this reason, by default it calls ui.write() 100 times with a single byte plus one newline byte, for 100 lines. To get the previous behavior, run `hg perfwrite --nlines=100000 --nitems=1 --item='Testing write performance' --batch-line`.

chg: fix typo

copies: implement __repr__ on branch_copies for debugging Differential Revision: https://phab.mercurial-scm.org/D8650

phases: remove useless lookup of repo[rev].rev() in _retractboundary changectx object is no longer needed thanks to a54ee130210a.

ui: replace `self._ferr` with identical `dest` Originally, it was part of a larger change that was abandoned. IMHO it makes the code slightly cleaner and saves one attribute access, so I decided to send it anyway instead of throwing it away.

absorb: make it clear what happens when no input Differential Revision: https://phab.mercurial-scm.org/D8643

tests: add hghave rule 'setprocname' to check if osutil.setprocname and use it setprocname is not present when we build a pure version. This leads to output changes in test-chg.t This should make test suite green on Python 2 pure build with chg. Differential Revision: https://phab.mercurial-scm.org/D8638

util: flush stderr explicitly after using warnings.warn() Due to some unknown reasons, when using chg with python3, the warnings.warn() output is not flushed. Fixes test-devel-warnings.t on py3 with chg.

phases: improve performance of _retractboundary The old version repeatedly converts nodes to revisions, which is a moderately expensive operation. Mapping all new changes once to revisions and back at the end reduces the time spend in _retractboundary during the unbundling of NetBSD's src from 67s to 17s. Differential Revision: https://phab.mercurial-scm.org/D8641