hgweb: validate WSGI environment dict The wsgiref.validate module contains useful functions for validating that various WSGI data structures are proper. This commit adds validation of the environment dict to our built-in HTTP server, which turns an HTTP request into an environment dict. The check discovered that we weren't always setting QUERY_STRING, which would cause the cgi module to fall back to sys.argv. So we change things to always set QUERY_STRING. The check passes on Python 2 and 3. Differential Revision: https://phab.mercurial-scm.org/D2731

hgweb: ensure all wsgi environment values are str Previously, we had a few entries that were bytes on Python 3. PEP-0333 states that all entries must be the native str type (bytes on Python 2, str on Python 3). This required a number of changes to hgweb_mod to unbreak things on Python 3. I suspect there still may be some regressions. I'm going to introduce a data structure that represents a parsed WSGI request in upcoming commits. This will hold bytes and will allow us to stop using raw literals throughout the WSGI code. Differential Revision: https://phab.mercurial-scm.org/D2730

wireproto: formalize permissions checking as part of protocol interface Per the inline comment desiring to formalize permissions checking in the protocol interface, we do that. I'm not convinced this is the best way to go about things. I would love for there to e.g. be a better exception for denoting permissions problems. But it does feel strictly better than snipping attributes on the proto instance. Differential Revision: https://phab.mercurial-scm.org/D2719

wireproto: declare permissions requirements in @wireprotocommand (API) With the security patches from 4.5.2 merged into default, we now have a per-command attribute defining what permissions are needed to run that command. We now have a richer @wireprotocommand that can be extended to record additional command metadata. So we port the permissions mechanism to be based on @wireprotocommand. .. api:: hgweb_mod.perms and wireproto.permissions have been removed. Wire protocol commands should declare their required permissions in the @wireprotocommand decorator. Differential Revision: https://phab.mercurial-scm.org/D2718

wireprotoserver: check permissions in main dispatch function The permissions checking code merged from stable is out of place in the refactored hgweb_mod module. This commit moves the main call to wireprotoserver. We still have some lingering code in hgweb_mod. This will get addressed later. Differential Revision: https://phab.mercurial-scm.org/D2717

wireprotoserver: check if command available before calling it The previous behavior was just plain wrong. I have no clue how it landed. My guess is a merge conflict resolution gone wrong on my end a few weeks ago. Differential Revision: https://phab.mercurial-scm.org/D2716

py3: drop encoding.strio() Its buffered nature makes TextIOWrapper unsuitable for temporarily wrapping bytes I/O.

ui: adjust Windows workaround to new _readline() code It's only needed when rawinput() is called. Also made it Py3 compatible.

ui: do not use rawinput() when we have to replace sys.stdin/stdout See the inline comment for why. The current Python3 hack doesn't work if more than one user inputs are expected because TextIOWrapper fills its internal buffer at the first read() request. Maybe we could write an unbuffered TextIOWrapper, but I don't want to make things more complicated. Instead, this patch reinvents raw_input(' ') of no readline support.

ui: do not try readline support if fin/fout aren't standard streams It's unlikely for a non-stdio stream to be a tty. Minimizing readline support makes it much simpler to work around the unicode input() function of Python 3. This also works on chg which duplicates client's tty to stdio fds.

util: add public isstdin/isstdout() functions

ui: add debug commands to test interactive prompt Interactive operations aren't easily covered by tests. So let's add commands to test them manually.

ui: inline util.bytesinput() into ui._readline() Prepares for rework of Python 3 support, which is currently broken due to read-ahead buffer of TextIOWrapper.

hgk: stop using util.bytesinput() to read a single line from stdin Appears that the stdio here is an IPC channel between hg and hgk (tk) processes, which shouldn't need a fancy readline support.

bookmarks: test for exchanging long bookmark names (issue5165) As far as I can tell a test for a long bookmark name never actually got added when this was fixed. Let's document that a 300 byte bookmark is something we're supporting (this patch started out life over a year ago as a way for me to validate the problem, and I recently found it.) I think the nonzero exits from the push operations are only because no new changesets get exchanged. Please correct me if I'm wrong on that. :) Differential Revision: https://phab.mercurial-scm.org/D2727

phabricator: follow-up phab auth improvements with backwards compat mode We'll rip this out before we ship 4.6, but this gives people a window to migrate. Differential Revision: https://phab.mercurial-scm.org/D2703

phabricator: specify API tokens per host, rather than per repo Differential Revision: https://phab.mercurial-scm.org/D1919

py3: drop b'' from generate-working-copy-states.py output I could make everything bytes, but decoding filename seemed easier and we don't have to worry about non-ascii filenames here.

py3: make test-commit-multiple.t byte-safe

py3: fix type of default username

py3: read/write plain lock file in binary mode A lock file shouldn't contain '\n', so this isn't a BC.

util: stop calling os.stat_float_times() It had Python-wide side effects, and it disappears in 3.7.0. As of this change, we're mostly working on 3.7.0b2. There are a few worrying failures, mostly around regular expressions, but we'll have to tackle those separately. Differential Revision: https://phab.mercurial-scm.org/D2697

cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime The latter is floating point by default, and we've been doing os.stat_float_times(False). Unfortunately, os.stat_float_times was removed between Python 3.7.0a1 and 3.7.0b2, so we have to stop using it. Differential Revision: https://phab.mercurial-scm.org/D2696

osutil: implement minimal __getitem__ compatibility on our custom listdir type We previously declined to do this, but the removal of the deprecated os.stat_float_times() method in Python 3.7 forces our hand. Differential Revision: https://phab.mercurial-scm.org/D2695

hgweb: adapt to socket._fileobject changes in Python 3 Differential Revision: https://phab.mercurial-scm.org/D2688

debugcommands: fix some %r output with bytestr() wrappers Almost fixes test-merge-tools.t. I think the remaining failure there is due to some overspecified tempfile names. Differential Revision: https://phab.mercurial-scm.org/D2675

tests: add test for issue 5494 but with --collapse This was not fixed, so the test case currently demonstrates the breakage. Differential Revision: https://phab.mercurial-scm.org/D2714

tests: .hg/merge is a directory, so use `test -d` This part of test-rebase-interrupts.t would have passed before the fix in a580b2d65ded (rebase: make sure merge state is cleaned up for no-op rebases (issue5494), 2017-05-18). Differential Revision: https://phab.mercurial-scm.org/D2713

rebase: only store collapse message once The message is determined by the user passing --message or --log when the rebase is started. There's no need to write it to a file for each rebased commit; writing it once at the start of the rebase is enough. Differential Revision: https://phab.mercurial-scm.org/D2712

rebase: collapse two nested if-conditions Also change the order since it feel to me like it's more about --collapse than it is about --keep. Differential Revision: https://phab.mercurial-scm.org/D2711