convert: add docstring on convert.subversion.geturl() The function is unusual for a bytes-handling function in Mercurial because it can’t handle arbitrary bytes. Therefore we should document this fact. Pointed out by Yuya Nishihara while reviewing e3b19004087a.

tests: use path inside test dir This will make the diff for the next patch less noisy.

convert: convert URLs to UTF-8 for Subversion Preamble: for comprehension, note that the `path` of geturl() would better be called `path_or_url` (the argument of the call of getsvn() is called `url`). For HTTP(S) URLs, the changes don’t make a difference, as they are restricted to ASCII. For file URLs, the reasoning is the same as for paths: we have to roundtrip with what Subversion is doing. When the locale encoding is ISO-8859-15, trying to convert a SVN repo `file:///tmp/a€` failed before like this: file:///tmp/a%A4 does not look like a Subversion repository to libsvn version 1.14.0 Decoding the path using the locale encoding can fail. In this case, we have to bail out, as Subversion won’t be able to do anything useful with the path.

convert: correctly convert paths to UTF-8 for Subversion The previous code using encoding.tolocal() only worked by chance in these situations: * The string is ASCII: The fast path was triggered and the string was returned unmodified. * The local encoding is UTF-8: The source and target encoding is the same. * The string is not valid UTF-8 and the native encoding is ISO-8859-1: If the string doesn’t decode using UTF-8, ISO-8859-1 is tried as a fallback. During `hg convert`, the local encoding is always UTF-8. The irony is that in this case, encoding.tolocal() behaves like what someone would expect the reverse function, encoding.fromlocal(), to do. When the locale encoding is ISO-8859-15, trying to convert a SVN repo `/tmp/a€` failed before like this: file:///tmp/a%C2%A4 does not look like a Subversion repository to libsvn version 1.14.0 The correct URL is `file:///tmp/a%E2%82%AC`. Unlike previously (with the ISO-8859-1 fallback), decoding the path using the locale encoding can fail. In this case, we have to bail out, as Subversion won’t be able to do anything useful with the path.

py3: pass URL as str Before the patch, HTTP(S) URLs were never recognized as a Subversion repository on Python 3.

convert: bail out in Subversion source if encountering non-ASCII HTTP(S) URL Before this patch, in the tested case, urllib raised `httplib.InvalidURL: URL can't contain control characters. '/\xff/!svn/ver/0/.svn' (found at least '\xff')`, which resulted in that the URL was never recognized as a Subversion repository. This patch adds a check that bails out if the URL contains non-ASCII characters. The warning is not overly user-friendly, but giving the user something to type into a search engine is definitively better than not explaining why the repository was not recognized. We could support non-ASCII chracters by quoting them before passing them to urllib. However, we would want to be compatible with what the `svn` command does, which converts the URL from the locale encoding to UTF-8, percent-encodes it and sends it to the server. If the locale encoding is not UTF-8, the behavior is IMHO not very intuitive, as the `svn` command may send different (percent-encoded) octets than what was passed on the console. Instead of copying this behavior, we better leave it forbidden.

run-tests: fix escapes with conditions Before this fix, escapes with conditions in tests failed like this on Python 3: $ $PYTHON -c 'from mercurial.utils.procutil import stdout; stdout.write(b"\xff")' - \xff (no-eol) (esc) (true !) + \xff (no-eol) (esc) The unicode_escape encoding decodes br'\xff' to u'\xff'. To convert the first 256 code points to bytes with the same ordinal, the latin-1 encoding must be used. Escapes without conditions already worked before on Python 3, but not through `el == l` a few lines below the changed line in run-tests.py. I didn’t investigate further.

convert: set LC_CTYPE around calls to Subversion bindings The Subversion bindings require that LC_CTYPE is set. However, we don’t want to set it all the time, as it changes the behavior of str methods on Python 2. The taken approach is hopefully fine-grained enough to not trigger any locale-specfic behavior of the str methods and coarse-grained enough to not clutter the code. Emulating the with-statement behavior in before() and after() should be safe, as after() is always called when before() is called. hgext.convert.hg takes a similar approach.

curses: do not initialize LC_ALL to user settings (issue6358) 701341f57ceb moved the setlocale() call to right before curses was used. This didn’t fully solve the problem it was supposed to solve (locale-dependent functions, like date formatting/parsing and str methods on Python 2), but only postponed it. Initializing LC_CTYPE seems to be sufficient for curses to work correctly. Therefore LC_CTYPE is set while curses is used and reset afterwards. Some locale-dependent str methods might behave differently on Python 2 while curses is used, but that shouldn’d be a problem.

hgweb: encode WSGI environment like OS environment Previously, the WSGI environment keys and values were encoded using latin-1. This resulted in a crash if a WSGI environment key or value could not be encoded using latin-1. On Unix, the OS environment is byte-based. Therefore we should do the reverse of what Python does for os.environ. On Windows, there’s no native byte-based OS environment. Therefore we should do the same as what mercurial.encoding does with the OS environment.

hgweb: deduplicate code A following patch will change the way keys and values are encoded. To reduce the diff, I’ve split off the uninteresting part.

share: provide a more useful text for hg help Differential Revision: https://phab.mercurial-scm.org/D8651

procutil: make recent fix for zombies compatible with py2 The fix in ed684a82e29b (procutil: always waiting on child processes to prevent zombies with 'hg serve', 2020-05-07) works only on Python 3 because it passes a `daemon` argument to `threading.Thread()`. Python 2 requires you to assign to the `.daemon` property instead. Python 3 also seems to support that, so this patch fixes the code by unconditionally using the old form. Differential Revision: https://phab.mercurial-scm.org/D8657

pycompat: fix crash when default locale is unknown Instead, fall back to the filesystem encoding if the default locale is unknown.

help: document meaning of '%' in graphlog output I added the feature in 14d0e89520a2 (graphlog: use '%' for other context in merge conflict, 2020-01-28), but I didn't think to look for documentation until today (because an internal user asked about it). Differential Revision: https://phab.mercurial-scm.org/D8649

py3: fix broken man page generation, it was generating `(default: NUL*)` `bytes(default)` was producing things like `(default: \x00)` when handed non-bytes values such as `1`, `10`, or `True`. The man page generation would apparently ignore these bytes and produce man pages that had the string `(default: )`. Test Plan: - Ran `cd doc; python3 gendoc.py "hg.1.gendoc"` and grepped for bad output - Ran `make deb`, extracted the deb, manually inspected `hg.1` file. Differential Revision: https://phab.mercurial-scm.org/D8639

py3: fix comparison between int and None If stop is None, the condition was always false on Python 2, as None compares smaller than ints. Therefore we make the condition false if stop is None.

py3: pass regex as bytes

py3: avoid using %r format on bytes Before the patch, the 'b' prefix appeared in the formatted string. Wrapping the bytes as pycompat.bytestr solves this problem. Eventually, I think that we should move away from using %r (like 975e517451a6 and 4d6019c0e0ef did), but that would change output of non-ASCII bytes on Python 2, so we can’t do it on the stable branch. Also, many places continue to use %r, so it would be a good idea to do the change all at once.

py3: use `%d` for int in % formatting On Python 3, `%s` is an alias to `%b`, which requires that the object implements `__bytes__()`, which is not the case for `int`.

py3: pass native string to urlreq.url2pathname() Of course, I’m not happy with the warning, but it’s better than crashing. Solving the problem properly is hard, and non-UTF-8 percent-encoded bytes in file URLs seem rare enough to block solving that all file URLs (even if not SVN-specific) will cause a crash.

py3: suppress DeprecationWarning about deprecated base64 module aliases base64.encodestring() / base64.decodestring() were renamed to base64.encodebytes() / base64.decodebytes() in Python 3. The old names still worked, but raised a DeprecationWarning.

py3: use `pycompat.ziplist()`

py3: use `%d` for int in % formatting On Python 3, `%s` is an alias to `%b`, which requires that the object implements `__bytes__()`, which is not the case for `int`.

py3: fix bytes iteration

py3: unbyteify arguments to warnings.filterwarnings() This fixes a crash when trying to import the convert extension on Python 3.

tests: adjust to the new format in pyflakes output According to the pyflakes' NEWS.rst, the default output format changed recently: 2.2.0 (2020-04-08) - Include column information in error messages So the lines now read: contrib/perf.py:149:15 undefined name 'xrange' mercurial/hgweb/server.py:427:13 undefined name 'reload' mercurial/util.py:2862:24 undefined name 'file' This is a graft of a similar fix that ended up on default. Differential Revision: https://phab.mercurial-scm.org/D8630

tests: consistently use pyflakes as a Python module We check availability of pyflakes as a module, and also running it for real as a module. Only fair to test filterpyflakes.py working correctly when using pyflakes as a module too. This is a graft of a similar fix that ended up on default. Differential Revision: https://phab.mercurial-scm.org/D8629

tests: skip pyflakes for mercurial/thirdparty/ The current version of pyflakes (2.2.0) correctly detects one issue: mercurial/thirdparty/selectors2.py:335:40 '...'.format(...) has unused arguments at position(s): 1 But we're not interested in fixing lint errors in third-party code, so we need to exclude at least selectors2.py. And in the discussion for this patch it was decided to just skip the entire thirdparty directory. This is a graft of a similar fix that ended up on default. Differential Revision: https://phab.mercurial-scm.org/D8628

zeroconf: fix non existant formatting in the vendored zeroconf module On Tue Mar 1st 2016 at 09:33:39 timeless decided to wrap long line in `hgext/zeroconf/Zeroconf.py`. Doing so, he fat fingered a "%w" instead of a "%s" in a string. %w does not exists, 4 year later, pyflakes (rightfully) complains about it. So I am fixing it. Differential Revision: https://phab.mercurial-scm.org/D8627

ignore: note debugignore on ignore man page It took me a long time to find debugignore. I found the ignore man page quickly. This change adds a debugging section to the ignore man page letting people know there is a debug command.

nodemap: fix validity checking when revlog is too short We cannot check the nodeid of a revision that is not even there. We add a simple fix and simple test.

resourceutil: fix location of line comments These comments slipped out of position when the sources where formatted with black in 2372284d9457.

rebase: avoid clobbering wdir() with --dry-run or --confirm (issue6291) See 56d3e0b499df for the source of adding originalwd to the list of things that cause wdir to be updated. That change didn't come with tests, and attempts to recreate the scenario described have thus far failed. Differential Revision: https://phab.mercurial-scm.org/D8489

tests: show that rebase --dry-run and --confirm wipeout uncommitted changes It looks like the carnage is limited to rebasing something that is not an ancestor of wdir(), as both of these abort in a preflight check for that case with a dirty working directory. Differential Revision: https://phab.mercurial-scm.org/D8488

Added signature for changeset 065704cbdbdb

Added tag 5.4.1 for changeset 065704cbdbdb

py3: update comment to account for Python 2 and Python 3 differences

py3: add warning about buffering behavior of pycompat.{stdout,stderr}

tests: fix indentation

graft: fix --base value to be saved in state file 'True' just works because it is treated as an integer revision '1' and only the truthiness of the basectx is important. If multiple source revisions were supported with --base, the resumed graft operation would go wrong.

flags: also test merge with executable bit removed This might catch more bug in the future. Differential Revision: https://phab.mercurial-scm.org/D8536

flags: also test the removal of the exec flag Differential Revision: https://phab.mercurial-scm.org/D8535

flags: read flag from dirstate/disk for workingcopyctx (issue5743) In 491855ea9d62, various piece of code are moved from committablectx to workingctx. The reason given is "These read from the dirstate, so they shouldn't be used in other subclasses." At least for `flags` this change introduce a bug, because the value flags end up being read from `_manifest` disregarding the actual state in the working copy (ie: on disk). When merging exec flag change with renames, this means a new files (the local content, renamed) is properly written on disk, with the right flags, but the flags part is later ignored when actually reading flags during merge. It is not clear to me why the `flags` function was moved, because the code does not actually hit the dirstate (the reason given in the changeset description). So I am moving it back to were it comes from and we use a simpler version of that code (that hit the dirstate everytime) in workingcopyctx. This fix the last know bug with merging rename and executable byte changes. Other similar bug might be lurking in 491855ea9d62, but I have not investigated them. Differential Revision: https://phab.mercurial-scm.org/D8534

flags: actually merge flags in simplemerge Since b86fc43e4b73, the local flag were blindly taken. This resulted in bug when rename are involved. exec flag change are now properly merged (when merged from the rename side). Another bug is affecting this when merging from the side without the rename. Differential Revision: https://phab.mercurial-scm.org/D8533

flags: add a test for merging exec flag change with rename and file change Changing the file activate other code path that also have bugs… There are two distinct bugs depending of which side of the merge you stand on. They both leading to exec flag loss. We add tests for both, the fix are coming in later changesets. Differential Revision: https://phab.mercurial-scm.org/D8532

flags: account for flag change when tracking rename relevant to merge There are some logic filtering rename to the one relevant to the merge. That logic was oblivious of flag change, leading to exec flag being dropped when merged with a renamed. There are two others bugs affecting this scenario. This patch fix the was where there is not modification involved except for the flag change. Fixes for the other bug are coming in later changesets. Differential Revision: https://phab.mercurial-scm.org/D8531

flags: also test merging a rename with and exec flag change This case is currently buggy and was not tested. This is probably a quite old regression. The next changeset fix this case. Move exec+rename related bug will gain a test later. To highlight the expected behavior the currently missing line are marked with (false !) and the bad one with (true !) note: we should probably gain explicit "test bool" for this usecases. Differential Revision: https://phab.mercurial-scm.org/D8530

flags: introduce explicit testing for merging change to exec flag It turns out that we do not seems to test the simple case for merging exec flag changes. More advanced case are test (merging exec flag without a common ancestors, merging with a symlink, etc…) but not the basic. We are about introduce various fixes to merging flag change across renames, having the most basic case tested first seems useful. note: We are only testing "adding" an exec flag here, not removing it. We introduce basic test on stable and will consolidate them on default. Differential Revision: https://phab.mercurial-scm.org/D8529

graft-state: save --base in graft's state, fixing bug with graft --continue Without this change, running graft --continue after grafting a merge commit using --base (and encountering conflicts) will output "skipping ungraftable merge revision" even though we specified a base in the initial graft command. Graft's improve behaviour is reflected in test-graft.t. Differential Revision: https://phab.mercurial-scm.org/D8578

setup: raise minimum Python version to 2.7.4 On older Python versions, Mercurial is not really usable because of https://bugs.python.org/issue10211. Recently someone reported a crash on the mailing list when running Mercurial on Python 2.7.3. There was consensus that fixing compatibility for a Python version more than 7 years old is not worth it. So, instead of making Mercurial crash with an obscure exception, this patch raises the minimum Python version to 2.7.4.

fsmonitor: coerce `clock` variable to byte-string (issue6321) Callers of `fsmonitor.state.setlastclock` pass their arguments wrapped in `pycompat.sysbytes` to ensure the value is a `bytes` on Python 3. However in `fsmonitor.poststatus.__call__`, if the return value of `getlastclock()` is `None`, we use the value of `fsmonitor.poststatus._startclock` instead, which is not converted to a byte string in the same manner. This commit converts the value of `startclock` to a byte string using `pycompat.sysbytes` in the constructor for `poststatus`, to avoid the "`str` + `bytes`" error from issue 6321. Differential Revision: https://phab.mercurial-scm.org/D8573

py3: change default priority and length used for sorting hooks to be compatible with python 3 The call to `sorted(hooks.values())` can on line 213 of hooks.py can raise when using python 3. For instance, when hooks.values is `[(0, 2, b'post-commit.check-status', b''), (None, None, b'changegroup.app-hooks', <object object at 0x7f5279885590>)]`, the error is `TypeError: '<' not supported between instances of 'NoneType' and 'int'` This fix keeps the same order that was used in python 2 without relying on comparison with None. Differential Revision: https://phab.mercurial-scm.org/D8527

relnotes: copy "next" to "5.4" and clear "next" This is the same thing as we've done for the previous few releases. Differential Revision: https://phab.mercurial-scm.org/D8546

dirstate: make sure the dirstate is loaded before the changelog (issue6303) Before this change, it was possible for the changelog to be loaded before the dirstate. If a transaction happens betwen the changelog and dirstate reading, the dirstate can up end poitning toward a revision not existing in the (olded) changelog. This lead to a warning. With this revision, we preload the dirstate parent before reading the changelog. This has a negligible performance impact on performance for all case we are tracking. Differential Revision: https://phab.mercurial-scm.org/D8528

rust-dirstatemap: don't read the dirstate when requesting parents A future patch for issue 6303 reveals a big performance regression in the Rust `DirstateMap` that reads the entire dirstate when requesting parents instead of the first 40 bytes. `perfdiscovery` gets a *significant* speedup (from 0.101 to 0.016) when applied against said patch. I'm assuming it has other performance benefits, but this is already a good enough win. Differential Revision: https://phab.mercurial-scm.org/D8513

py3: fix exception in pull when several things happen to a bookmark Specifically, when `changes` is: [(b'@upstream-committed', None, <function ui.status>, b'updating bookmark @upstream-committed\n'), (b'@upstream-committed', binary-node, <function ui.warn>, b'divergent bookmark @ stored as @upstream-committed\n')] sorting the list raises: TypeError: '<' not supported between instances of 'bytes' and 'NoneType' Differential Revision: https://phab.mercurial-scm.org/D8523

fastexport: adjust output to be more canonical For time zones, git doesn't consider +0 and -0 the same timezone, so use the former canonically. Add a test case to ensure that non-UTC offsets are handled correctly. The real name part of the committer name is normally not quoted, so don't enforce that. Differential Revision: https://phab.mercurial-scm.org/D8522

bash_completion: do not use aliased hg if it sources a script (issue6308) I have an alias that sources a script around hg. Mercurial's bash_completion script tries to use this as its main hg binary. But sourcing a wrapper breaks Bash's completion. So this patch disables using the alias as the hg binary if it starts with "source ". Alias resolution was introduced in rev 191ab08e7099 for users with "alias hg='hg --some_opts'". See https://www.mercurial-scm.org/repo/hg/rev/191ab08e7099

demandimport: fix compatibility with meta path finders w/o find_spec() method Meta path finders got a find_spec() method in Python version 3.4. The sys.meta_path documentation says that the deprecated find_module() method is used as a fallback. Setuptool’s VendorImporter still doesn’t have the find_spec() method, which resulted in a crash e.g. within a virtual environment. For reference, I opened an issue for that: https://github.com/pypa/setuptools/issues/2104. An alternative implementation would have been to implement a wrapper for find_module() itself and raise an AttributeError when accessing find_spec() if the wrapped finder doesn’t have it.