merge with stable There were a handful of merge conflicts in the wire protocol code due to significant refactoring in default. When resolving the conflicts, I tried to produce the minimal number of changes to make the incoming security patches work with the new code. I will send some follow-up commits to get the security patches better integrated into default.

sslutil: some more forcebytes() on some exception messages At this point, test-https.t no longer dumps tracebacks everywhere. Instead, we get some results that look like we're not adequately finding things in hg's configuration, which should be manageable (if somewhat annoying to find and fix.) Differential Revision: https://phab.mercurial-scm.org/D2690

sslutil: sslcontext needs the cipher name as a sysstr Differential Revision: https://phab.mercurial-scm.org/D2689

sslutil: lots of unicode/bytes cleanup In general, we handle hostnames as bytes, except where Python forces them to be unicodes. This fixes all the tracebacks I was seeing in test-https.t, but there's still some ECONNRESET weirdness that I can't hunt down... Differential Revision: https://phab.mercurial-scm.org/D2687

debugwireproto: handle unimplemented util.poll() for Windows This is the same logic used in sshpeer.doublepipe. It doesn't completely fix test-ssh-proto{,-unbundle}.t ("read(-1) -> X" is changed to "read(X) -> X", the order of some lines are changed, and abort messages seem to be missing), but it cuts down a ton on the failure spew.

py3: byte-stringify test-blackbox.t

py3: byte-stringify blackbox-readonly-dispatch.py # skip-blame because just adding some b''

py3: make blackbox-readonly-dispatch.py use ui instead of print()

py3: fix int formatting of "incoming changes" log

largefiles: use %d instead of %s to process ints Differential Revision: https://phab.mercurial-scm.org/D2677

transaction: fix an error string with bytestr() on a repr()d value Fixes test-rollback.t on Python 3. Differential Revision: https://phab.mercurial-scm.org/D2674

py3: work around comparison between int and None in tagmerge

py3: do not mutate dict while iterating in tagmerge

py3: fix type of ui.configitems(ignoresub=True) result

py3: don't use str() to stringify pushloc

py3: byte-stringify test-config.t and test-config-env.py

py3: use startswith() instead of slicing to detect leading whitespace

archival: use py3 friendly replacements for chr() and long() Differential Revision: https://phab.mercurial-scm.org/D2673

archival: ensure file mode for gzipfile is sysstr Differential Revision: https://phab.mercurial-scm.org/D2672

archival: fix a missing r'' on a kwargs check # skip-blame just an r prefix Differential Revision: https://phab.mercurial-scm.org/D2671

py3: more passing tests (ten this time) Differential Revision: https://phab.mercurial-scm.org/D2670

util: fix unsafe url abort with bytestr() on url Differential Revision: https://phab.mercurial-scm.org/D2669

xdiff: fix builds on Windows This works on my ancient Fedora system too, without warnings. There are, however, warnings about various 64 to 32 bit conversions on Windows that need to be examined.

py3: use bytes() instead of str() Differential Revision: https://phab.mercurial-scm.org/D2665

py3: use b"%d" instead of str() to convert integers to bytes Differential Revision: https://phab.mercurial-scm.org/D2664

py3: use util.forcebytestr to convert testedwith value to bytes Bad extensions can put anything in testedwith so we should use util.forcebytestr here. Differential Revision: https://phab.mercurial-scm.org/D2663

py3: use pycompat.bytestr instead of str Differential Revision: https://phab.mercurial-scm.org/D2648

setdiscovery: avoid a Yoda condition Differential Revision: https://phab.mercurial-scm.org/D2646

setdiscovery: remove unnecessary sample size limiting Both _takequicksample() and _takefullsample() already limit their result to the request size, so there's no need to let the caller do that again. Differential Revision: https://phab.mercurial-scm.org/D2645

setdiscovery: remove initialsamplesize from a condition It seems more direct to compare the actual sample size. That way we can change the sample taken earlier in the code without breaking the condition. Differential Revision: https://phab.mercurial-scm.org/D2644

setdiscovery: back out changeset 5cfdf6137af8 (issue5809) As explained in the bug report, this commit caused a performance regression. The problem occurs when the local repo has very many heads. Before 5cfdf6137af8, we used to get the remote's list of heads and if these heads mostly overlapped with the local repo's heads, we would mark these common heads as common, which would greatly reduce the size of the set of undecided nodes. Note that a similar problem existed before 5cfdf6137af8: If the local repo had very many heads and the server just had a few (or many heads from a disjoint set), we would do the same kind of slow discovery as we would with 5cfdf6137af8 in the case where local and remote repos share a large set of common nodes. For now, we just back out 5cfdf6137af8. We should improve the discovery in the "local has many heads, remote has few heads" case, but let's do that after backing this out. Differential Revision: https://phab.mercurial-scm.org/D2643

hgweb: fix up trailing slash detection on Python 3 Fixes a couple of hgweb tests. Differential Revision: https://phab.mercurial-scm.org/D2661

hgweb: convert req.form to bytes for all keys and values This is just going to be a lot cleaner for our internals. Differential Revision: https://phab.mercurial-scm.org/D2660

templater: show repr of string we're rejecting I feel like this should make it a little easier to hunt down problems. Differential Revision: https://phab.mercurial-scm.org/D2659

webutil: some %d instead of %s love on ints Differential Revision: https://phab.mercurial-scm.org/D2658

py3: whitelist three more cases Differential Revision: https://phab.mercurial-scm.org/D2657

archival: our filenames are bytes, not strs Differential Revision: https://phab.mercurial-scm.org/D2656

archival: tar file modes need to be sysstrs Differential Revision: https://phab.mercurial-scm.org/D2655

archival: fsdecode paths before passing to tar or zip objects Both of these traffic in unicodes for filenames on Python 3, and inspection of the tarfile module shows that it uses the filesystem encoding, so fsdecode is the right choice. Differential Revision: https://phab.mercurial-scm.org/D2654

py3: add b'' prefixes in tests/test-minirst.py # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D2653

py3: make sure __repr__ returns a str # skip-blame because just r'' prefix Differential Revision: https://phab.mercurial-scm.org/D2652

py3: make sure regular expressions are bytes # skip-blame because just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D2651

py3: use bytes instead of str to make sure we use bytes internally Differential Revision: https://phab.mercurial-scm.org/D2650

py3: use util.forcebytestr instead of str to convert error messages Differential Revision: https://phab.mercurial-scm.org/D2649

lock: block signal interrupt while making a lock file On Windows where symlink isn't supported, util.makelock() could leave an empty file if interrupted immediately after os.open(). This empty lock never dies as it has no process id recorded. ld = os.open(pathname, os.O_CREAT | os.O_WRONLY | os.O_EXCL) # an interrupt may occur here os.write(ld, info) os.close(ld) This was a long-standing bug of TortoiseHg which runs a command-server and kills it by CTRL_C_EVENT, reported by random Windows users. https://bitbucket.org/tortoisehg/thg/issues/4873/#comment-43591129 At first, I tried to fix makelock() to clean up a stale lock file, which turned out to be hard because any instructions may be interrupted by a signal. ld = None try: # CALL_FUNCTION # os.open(...) # an interrupt may occur here # STORE_FAST # ld = ... ld = os.open(pathname, os.O_CREAT | os.O_WRONLY | os.O_EXCL) os.write(ld, info) ... return True except: if ld: ... os.unlink(pathname) return False So I decided to block signals by temporarily replacing the signal handlers so makelcok() and held = 1 will never be interrupted. Many thanks to Fernando Najera for investigating the issue.

fuzz: add some more docs about building/running fuzzers Differential Revision: https://phab.mercurial-scm.org/D2635

util: also silence py3 warnings from codec module Fixes warnings like this: + mercurial/util.py:2446: DeprecationWarning: invalid escape sequence '\d' + return codecs.escape_decode(s)[0] Differential Revision: https://phab.mercurial-scm.org/D2642

docs: small fixes for profiling.nested and the overall description - profiling.nested defaults to 0, not 5 - profiling is not always done with lsprof Differential Revision: https://phab.mercurial-scm.org/D2641

scmutil: fix oversight in b76248e51605c6 where I forgot to use msg Thanks to Yuya for spotting my mistake. Differential Revision: https://phab.mercurial-scm.org/D2636

hghave: remove unused "as ex" in exception block I overlooked this when removing a debug print in another change. Differential Revision: https://phab.mercurial-scm.org/D2637

tests: port test-log to Python 3 Required some porting to >>> inline Python instead of using heredocs into $PYTHON. Differential Revision: https://phab.mercurial-scm.org/D2621

py3: make gettext domain a system string

templater: fix position of terminator character in error message Since a template expression starts after the '{' character, the expression should be considered to end immediately before the terminator '{'.

merge with stable

revsetlang: add a hint for more useful parse errors This logic is largely based on the similar logic added to template error messages in D2608 and D2609, but with a few tweaks based on how revsets actually work. Differential Revision: https://phab.mercurial-scm.org/D2619

setup: ignore extension load failures when finding working hg Previously, `make local` would fail if any extension was not properly loading. Differential Revision: https://phab.mercurial-scm.org/D2589

profile: colorize output on Windows Previously, the ANSI codes were printed to the screen, throwing off the alignment. We could probably do this unconditionally, but it's kind of a hack, so I figured I'd limit the scope.

dispatch: don't clamp the range of the exit code twice We already limit the range to (0, 255) in the call to sys.exit(). The duplicated operation can't possibly be hurting us, but let's clean it up to avoid confusion.

bdiff: avoid pointer arithmetic on void* MSVC 2008 complains: mercurial/cext/bdiff.c(106) : error C2036: 'void *' : unknown size mercurial/cext/bdiff.c(107) : error C2036: 'void *' : unknown size Maybe it's a gcc extension? https://stackoverflow.com/questions/37460579/error-c2036-void-unknown-size

fuzz: add a quick README to try and document how to test new fuzzers Differential Revision: https://phab.mercurial-scm.org/D2633