py3: fix comparison between int and None If stop is None, the condition was always false on Python 2, as None compares smaller than ints. Therefore we make the condition false if stop is None.

py3: pass regex as bytes

py3: avoid using %r format on bytes Before the patch, the 'b' prefix appeared in the formatted string. Wrapping the bytes as pycompat.bytestr solves this problem. Eventually, I think that we should move away from using %r (like 975e517451a6 and 4d6019c0e0ef did), but that would change output of non-ASCII bytes on Python 2, so we can’t do it on the stable branch. Also, many places continue to use %r, so it would be a good idea to do the change all at once.

py3: use `%d` for int in % formatting On Python 3, `%s` is an alias to `%b`, which requires that the object implements `__bytes__()`, which is not the case for `int`.

py3: pass native string to urlreq.url2pathname() Of course, I’m not happy with the warning, but it’s better than crashing. Solving the problem properly is hard, and non-UTF-8 percent-encoded bytes in file URLs seem rare enough to block solving that all file URLs (even if not SVN-specific) will cause a crash.

py3: suppress DeprecationWarning about deprecated base64 module aliases base64.encodestring() / base64.decodestring() were renamed to base64.encodebytes() / base64.decodebytes() in Python 3. The old names still worked, but raised a DeprecationWarning.

py3: use `pycompat.ziplist()`

py3: use `%d` for int in % formatting On Python 3, `%s` is an alias to `%b`, which requires that the object implements `__bytes__()`, which is not the case for `int`.

py3: fix bytes iteration

py3: unbyteify arguments to warnings.filterwarnings() This fixes a crash when trying to import the convert extension on Python 3.

fuzz: add config knob for PYTHON_CONFIG_FLAGS I'll clean this up once we get oss-fuzz to use Python 3.8 instead of 2.7, but for now we need a way to evolve the flags passed to python-config in lockstep with the Python version. Yuck. Differential Revision: https://phab.mercurial-scm.org/D8637

pyutil: this has taken so long to fix, I'm using 3.8 now Differential Revision: https://phab.mercurial-scm.org/D8636

merge with stable

tests: adjust to the new format in pyflakes output According to the pyflakes' NEWS.rst, the default output format changed recently: 2.2.0 (2020-04-08) - Include column information in error messages So the lines now read: contrib/perf.py:149:15 undefined name 'xrange' mercurial/hgweb/server.py:427:13 undefined name 'reload' mercurial/util.py:2862:24 undefined name 'file' This is a graft of a similar fix that ended up on default. Differential Revision: https://phab.mercurial-scm.org/D8630

tests: consistently use pyflakes as a Python module We check availability of pyflakes as a module, and also running it for real as a module. Only fair to test filterpyflakes.py working correctly when using pyflakes as a module too. This is a graft of a similar fix that ended up on default. Differential Revision: https://phab.mercurial-scm.org/D8629

tests: skip pyflakes for mercurial/thirdparty/ The current version of pyflakes (2.2.0) correctly detects one issue: mercurial/thirdparty/selectors2.py:335:40 '...'.format(...) has unused arguments at position(s): 1 But we're not interested in fixing lint errors in third-party code, so we need to exclude at least selectors2.py. And in the discussion for this patch it was decided to just skip the entire thirdparty directory. This is a graft of a similar fix that ended up on default. Differential Revision: https://phab.mercurial-scm.org/D8628

zeroconf: fix non existant formatting in the vendored zeroconf module On Tue Mar 1st 2016 at 09:33:39 timeless decided to wrap long line in `hgext/zeroconf/Zeroconf.py`. Doing so, he fat fingered a "%w" instead of a "%s" in a string. %w does not exists, 4 year later, pyflakes (rightfully) complains about it. So I am fixing it. Differential Revision: https://phab.mercurial-scm.org/D8627

ignore: note debugignore on ignore man page It took me a long time to find debugignore. I found the ignore man page quickly. This change adds a debugging section to the ignore man page letting people know there is a debug command.

nodemap: fix validity checking when revlog is too short We cannot check the nodeid of a revision that is not even there. We add a simple fix and simple test.

tests: remove unused creation of file and outdated text It was forgotten to remove this in fb0de0bcd297.

tests: skip pyflakes for mercurial/thirdparty/ The current version of pyflakes (2.2.0) correctly detects one issue: mercurial/thirdparty/selectors2.py:335:40 '...'.format(...) has unused arguments at position(s): 1 But we're not interested in fixing lint errors in third-party code, so we need to exclude at least selectors2.py. And in the discussion for this patch it was decided to just skip the entire thirdparty directory. Differential Revision: https://phab.mercurial-scm.org/D8619

git: add debug logging when there's a mismatch in the cached heads list The dag rebuild can be expensive, so let's try and avoid bugs where it transparently rebuilds all the time for no reason. This would have prevented the issue fixed in D8622. Differential Revision: https://phab.mercurial-scm.org/D8625

py3: make stdout line-buffered if connected to a TTY Status messages that are to be shown on the terminal should be written to the file descriptor before anything further is done, to keep the user updated. One common way to achieve this is to make stdout line-buffered if it is connected to a TTY. This is done on Python 2 (except on Windows, where libc, which the CPython 2 streams depend on, does not properly support this). Python 3 rolls it own I/O streams. On Python 3, buffered binary streams can't be set line-buffered. The previous code (added in 227ba1afcb65) incorrectly assumed that on Python 3, pycompat.stdout (sys.stdout.buffer) is already line-buffered. However the interpreter initializes it with a block-buffered stream or an unbuffered stream (when the -u option or the PYTHONUNBUFFERED environment variable is set), never with a line-buffered stream. One example where the current behavior is unacceptable is when running `hg pull https://www.mercurial-scm.org/repo/hg` on Python 3, where the line "pulling from https://www.mercurial-scm.org/repo/hg" does not appear on the terminal before the hg process blocks while waiting for the server. Various approaches to fix this problem are possible, including: 1. Weaken the contract of procutil.stdout to not give any guarantees about buffering behavior. In this case, users of procutil.stdout need to be changed to do enough flushes. In particular, 1. either ui must insert enough flushes for ui.write() and friends, or 2. ui.write() and friends get split into flushing and fully buffered methods, or 3. users of ui.write() and friends must flush explicitly. 2. Make stdout unbuffered. 3. Make stdout line-buffered. Since Python 3 does not natively support that for binary streams, we must implement it ourselves. (2.) is problematic because using unbuffered I/O changes the performance characteristics significantly compared to line-buffered (which is used on Python 2) and this would be a regression. (1.2.) and (1.3) are a substantial amount of work. It’s unclear whether the added complexity would be justified, given that raw performance doesn’t matter that much when writing to a terminal much faster than the user could read it. (1.1.) pushes complexity into the ui class instead of separating the concern of how stdout is buffered. Other users of procutil.stdout would still need to take care of the flushes. This patch implements (3.). The general performance considerations are very similar to (1.1.). The extra method invocation and method forwarding add a little more overhead if the class is used. In exchange, it doesn’t add overhead if not used. For the benchmarks, I compared the previous implementation (incorrect on Python 3), (1.1.), (3.) and (2.). The command was chosen so that the streams were configured as if they were writing to a TTY, but actually write to a pager, which is also the default: HGRCPATH=/dev/null python3 ./hg --cwd ~/vcs/mozilla-central --time --pager yes --config pager.pager='cat > /dev/null' status --all previous: time: real 7.880 secs (user 7.290+0.050 sys 0.580+0.170) time: real 7.830 secs (user 7.220+0.070 sys 0.590+0.140) time: real 7.800 secs (user 7.210+0.050 sys 0.570+0.170) (1.1.) using Yuya Nishihara’s patch: time: real 9.860 secs (user 8.670+0.350 sys 1.160+0.830) time: real 9.540 secs (user 8.430+0.370 sys 1.100+0.770) time: real 9.830 secs (user 8.630+0.370 sys 1.180+0.840) (3.) using this patch: time: real 9.580 secs (user 8.480+0.350 sys 1.090+0.770) time: real 9.670 secs (user 8.480+0.330 sys 1.170+0.860) time: real 9.640 secs (user 8.500+0.350 sys 1.130+0.810) (2.) using a previous patch by me: time: real 10.480 secs (user 8.850+0.720 sys 1.590+1.500) time: real 10.490 secs (user 8.750+0.750 sys 1.710+1.470) time: real 10.240 secs (user 8.600+0.700 sys 1.590+1.510) As expected, there’s no difference on Python 2, as exactly the same code paths are used: previous: time: real 6.950 secs (user 5.870+0.330 sys 1.070+0.770) time: real 7.040 secs (user 6.040+0.360 sys 0.980+0.750) time: real 7.070 secs (user 5.950+0.360 sys 1.100+0.760) this patch: time: real 7.010 secs (user 5.900+0.390 sys 1.070+0.730) time: real 7.000 secs (user 5.850+0.350 sys 1.120+0.760) time: real 7.000 secs (user 5.790+0.380 sys 1.170+0.710)

simplemerge: rewrite flag merging loop as expression I feel binary operations are more readable.

simplemerge: leverage pycompat function to convert byte string to set

simplemerge: fix function name that tests if ctx is not null revision

git: decode node IDs back into Python strings (issue6349) db.text_factory = bytes, so the database contains only strings. The object IDs we get from pygit2 are Python strings. b'foo' != 'foo' This change allows the "don't reindex" optimization to work by allowing the "cur_cache_heads == cache_heads" comparison a few lines down to succeed. Differential Revision: https://phab.mercurial-scm.org/D8622

phabricator: make it clear what happen when no response Differential Revision: https://phab.mercurial-scm.org/D8621

tests: make it clear what happen when no response entered Differential Revision: https://phab.mercurial-scm.org/D8620

localrepo: handle ValueError during repository opening Python 3.8 can raise ValueError on attempt of an I/O operation against an illegal path. This was causing test-remotefilelog-gc.t to fail on Python 3.8. This commit teaches repository opening to handle ValueError and re-raise an Abort on failure. An arguably better solution would be to implement this logic in the vfs layer. But that seems like a bag of worms and I don't want to go down that rabbit hole. Until users report uncaught ValueError exceptions in the wild, I think it is fine to patch this at the only occurrence our test harness is finding it. Differential Revision: https://phab.mercurial-scm.org/D7944