hgweb: remove support for short query string based aliases (BC) Form data exposed by hgweb is post-processed to expand certain shortcuts. For example, URLs with "?cs=@" is essentially expanded to "?cmd=changeset&node=@". And the URL router treats this the same as "/changeset/@". These shortcuts were initially added in 2005 in 34cb3957d875 and 964baa35faf8. They have rarely been touched in the last decade (just moving code around a bit). We have almost no test coverage of this feature. AFAICT no templates reference URLs of this form. I even looked at the initial version of paper and coal from ~2008 and they use the "/command/params" URL form and not these shortcuts. Furthermore, I couldn't even get some shortcuts to work! For example, "?sl=@" attempts to do a revision search instead of showing shortlog starting at revision @. Maybe I'm just doing it wrong? Because this is ancient, mostly untested code, there is a migration path to something better, and because anyone passionate enough to preserve URLs can install URL redirects, let's nuke the feature. .. bc:: Query string shorts in hgweb like ``?cs=@`` have been removed. Use URLs of the form ``/:cmd`` instead. Differential Revision: https://phab.mercurial-scm.org/D2773

hgweb: remove support for POST form data (BC) Previously, we called out to cgi.parse(), which for POST requests parsed multipart/form-data and application/x-www-form-urlencoded Content-Type requests for form data, combined it with query string parameters, returned a union of the values. As far as I know, nothing in Mercurial actually uses this mechanism to submit data to the HTTP server. The wire protocol has its own mechanism for passing parameters. And the web interface only does GET requests. Removing support for parsing POST data doesn't break any tests. Another reason to not like this feature is that cgi.parse() may modify the QUERY_STRING environment variable as a side-effect. In addition, it merges both POST data and the query string into one data structure. This prevents consumers from knowing whether a variable came from the query string or POST data. That can matter for some operations. I suspect we use cgi.parse() because back when this code was initially implemented, it was the function that was readily available. In other words, I don't think there was conscious choice to support POST data: we just got it because cgi.parse() supported it. Since nothing uses the feature and it is untested, let's remove support for parsing POST form data. We can add it back in easily enough if we need it in the future. .. bc:: Hgweb no longer reads form data in POST requests from multipart/form-data and application/x-www-form-urlencoded requests. Arguments should be specified as URL path components or in the query string in the URL instead. Differential Revision: https://phab.mercurial-scm.org/D2774

hgweb: expose input stream on parsed WSGI request object Our next step towards moving away from wsgirequest to our newer, friendlier parsedrequest type is input stream access. This commit exposes the input stream on the instance. Consumers in the HTTP protocol server switch to it. Because there were very few consumers of the input stream, we stopped storing a reference to the input stream on wsgirequest directly. All access now goes through parsedrequest. However, wsgirequest still may read from this stream as part of cgi.parse(). So we still need to create the stream from wsgirequest. Differential Revision: https://phab.mercurial-scm.org/D2771

hgweb: make parsedrequest part of wsgirequest This is kind of ugly. But an upcoming commit will teach parsedrequest about the input stream. Because the input stream is global state and can't be accessed without side-effects, we need to take actions to ensure that multiple consumers don't read from it independently. The easiest way to do this is for one object to hold a reference to both items having access to the input stream so that when a copy is made, we can remove the attribute from the other instance. So we create our parsed request instance from the wsgirequest constructor and hold a reference to it there. This is better than our new type holding a reference to wsgirequest because all the code for managing access will be temporary and we shouldn't pollute parsedrequest with this ugly history. Differential Revision: https://phab.mercurial-scm.org/D2770

hgweb: refactor the request draining code The previous code for draining was only invoked in a few places in the wire protocol. Behavior wasn't consist. Furthermore, it was difficult to reason about. With us converting the input stream to a capped reader, it is now safe to always drain the input stream when its size is known because we can never overrun the input and read into the next HTTP request. The only question is "should we?" This commit changes the draining code so every request is examined. Draining now kicks in for a few requests where it wouldn't before. But I think the code is sufficiently restricted so the behavior is safe. Possibly the most dangerous part of this code is the issuing of Connection: close for POST and PUT requests that don't have a Content-Length. I don't think there are any such uses in our WSGI application, so this should be safe. In the near future, I plan to significantly refactor the WSGI response handling. I anticipate this code evolving a bit. So any minor regressions around draining or connection closing behavior might be fixed as a result of that work. All tests pass with this change. That scares me a bit because it means we are lacking low-level tests for the HTTP protocol. Differential Revision: https://phab.mercurial-scm.org/D2769

hgweb: use a capped reader for WSGI input stream Per PEP 3333, the input stream from WSGI should respect EOF and prevent reads past the end of the request body. However, not all WSGI servers guarantee this. Notably, our BaseHTTPServer based built-in HTTP server doesn't. Instead, it exposes the raw socket and you can read() from it all you want, getting the connection in a bad state by doing so. We have a "cappedreader" utility class that proxies a file object and prevents reading past a limit. This commit converts the WSGI input stream into a capped reader when the input length is advertised via Content-Length headers. "cappedreader" only exposes a read() method. PEP 3333 states that the input stream MUST also support readline(), readlines(hint), and __iter__(). However, since our WSGI application code only calls read() and since we're not manipulating the stream exposed by the WSGI server, we're not violating the spec here. Differential Revision: https://phab.mercurial-scm.org/D2768

hgweb: document continuereader Differential Revision: https://phab.mercurial-scm.org/D2767

hgweb: remove wsgirequest.__iter__ This was added in d0db3462d568 in 2006. I can't find a justification for this method in PEP 3333. I suspect we were originally intending to use this type as the WSGI application (which should be iterable)? The tests all pass without this method. So let's nuke it. Differential Revision: https://phab.mercurial-scm.org/D2749

hgweb: remove wsgirequest.read() This was just a proxy to self.inp.read(). This method serves little value. Let's nuke it. Callers in the wire protocol server have been updated accordingly. Differential Revision: https://phab.mercurial-scm.org/D2748

hgweb: remove unused methods on wsgirequest writelines() isn't used in our code base. close() was a no-op. It is an optional method per PEP 3333. My eventual goal is to kill the wsgirequest class, hence why I'm removing code. Differential Revision: https://phab.mercurial-scm.org/D2747

wireprotoserver: remove unused argument from _handlehttperror() Differential Revision: https://phab.mercurial-scm.org/D2746

hgweb: store and use request method on parsed request PEP 3333 says that REQUEST_METHOD is always defined. Differential Revision: https://phab.mercurial-scm.org/D2745

hgweb: handle CONTENT_LENGTH PEP 3333 says CONTENT_LENGTH may be set. I /think/ WSGI servers are allowed to invent this key even if the client didn't send it. We had code in wireprotoserver looking for this key. So let's just automagically convert this key to an HTTP request header when parsing the request. Differential Revision: https://phab.mercurial-scm.org/D2744

wireprotoserver: access headers through parsed request Now that we can access headers via the parsed request object, let's do that. Since the new object uses bytes, hyphens, and is case-insensitive, a bit of code around normalizing values has been removed. I think the new code is much more intuitive because it more closely matches what is going out over the wire. Differential Revision: https://phab.mercurial-scm.org/D2743

debugwireproto: close the write end before consuming all available data And make it read all available data deterministically. Otherwise util.poll() may deadlock because both stdout and stderr could have no data. Spotted by the next patch which removes stderr from the fds.

graft: check for missing revision first before scanning working copy Differential Revision: https://phab.mercurial-scm.org/D2753

hook: ensure stderr is flushed when an exception is raised, for test stability Windows has had issues with output order in test-ssh-proto-unbundle.t[1] since it was created a few weeks ago. Each of the problems occurred when an exception was thrown out of the hook. Now the only thing blocking D2720 is the fact that the "abort: ..." lines on stderr are totally AWOL. I have no idea where there are. [1] https://buildbot.mercurial-scm.org/builders/Win7%20x86_64%20hg%20tests/builds/541/steps/run-tests.py%20%28python%202.7.13%29/logs/stdio

wireproto: raise ProgrammingError instead of Abort This isn't a user-facing error and can only be caused by bad Python code. Thanks to Yuya for spotting this. Differential Revision: https://phab.mercurial-scm.org/D2777

py3: make test-commit-interactive.t byte-safe

py3: open patch file in binary mode and convert eol manually Here we don't introduce a reader wrapper since it wouldn't be easy to make read(n) handle partial data and length correctly.

py3: wrap file object to write patch in native eol preserving byte-ness

py3: drop b'' from debug message "moving bookmarks"

py3: use r'' instead of sysstr('') to get around code transformer Fewer function calls should be better.

ui: remove any combinations of CR|LF from prompt response On Windows, we have to accept both CR+LF and LF. This patch simply makes any trailing CRs and LFs removed from a user input instead of doing stricter parsing, as an input must be a readable text.

sshpeer: check pipe validity before forwarding output from it After the previous fix, fileobjectproxy._observedcall() (called when win32.peekpipe() accesses .fileno) started exploding. With this fix, similar checks are needed inside debugwireproto(). Since that is hardcoded to not use os.devnull, IDK if those are worth fixing.

util: forward __bool__()/__nonzero__() on fileobjectproxy In trying to debug the Windows process hang in D2720, I changed the stderr pipe to the peer to be os.devnull instead. That caused sshpeer._cleanuppipes()[1] to explode, complaining NoneType has no __iter__ attribute, even though the previous line checked for None. [1] https://www.mercurial-scm.org/repo/hg/file/b434965f984e/mercurial/sshpeer.py#l133

py3: fix slicing of bisect label in templatefilters.shortbisect()

templatefilters: inline hbisect.shortlabel() It's pretty simple. I don't think the business logic has to be placed in hbisect.py.

py3: make test-bisect.t bytes-safe

py3: fix integer formatting in bisect error

py3: silence f.write() in test-annotate.t

xdiff: resolve signed unsigned comparison warning Since the value won't be changed inside the code (because context lines feature was removed by D2705), let's just remove the variable and inline the 0 value. The code might be potentially further simplified. But I'd like to make sure correctness is easily verifiable in this patch. Differential Revision: https://phab.mercurial-scm.org/D2766

xdiff: use int64 for hash table size Follow-up of the previous "long" -> "int64" change. Now xdiff only uses int for return values and small integers (ex. booleans, shifting score, bits in hash table size, etc) so it should be able to handle large input. Differential Revision: https://phab.mercurial-scm.org/D2765

xdiff: remove unused xpp and xecfg parameters They are unused. Thus removed. Differential Revision: https://phab.mercurial-scm.org/D2764

xdiff: remove unused flags parameter After D2683, the flags parameter in some functions is no longer needed. Thus removed. Differential Revision: https://phab.mercurial-scm.org/D2763

xdiff: replace {unsigned ,}long with {u,}int64_t MSVC treats "long" as 4-byte. That could cause overflows since the xdiff code uses "long" in places where "size_t" or "ssize_t" should be used. Let's use explicit 8 byte integers to avoid FWIW git avoids that overflow by limiting diff size to 1GB [1]. After examining the code, I think the remaining risk (the use of "int") is low since "int" is only used for return values and hash table size. Although a wrong hash table size would not affect the correctness of the code, but that could make the code extremely slow. The next patch will change hash table size to 8-byte integer so the 1GB limit is unlikely needed. This patch was done by using `sed`. [1]: https://github.com/git/git/commit/dcd1742e56ebb944c4ff62346da4548e1e3be67 Differential Revision: https://phab.mercurial-scm.org/D2762

xdiff: add comments for fields in xdfile_t This makes the related code easier to understand. Differential Revision: https://phab.mercurial-scm.org/D2685

xdiff: add a preprocessing step that trims files xdiff has a `xdl_trim_ends` step that removes common lines, unmatchable lines. That is in theory good, but happens too late - after splitting, hashing, and adjusting the hash values so they are unique. Those splitting, hashing and adjusting hash values steps could have noticeable overhead. Diffing two large files with minor (one-line-ish) changes are not uncommon. In that case, the raw performance of those preparation steps seriously matter. Even allocating an O(N) array and storing line offsets to it is expensive. Therefore my previous attempts [1] [2] cannot be good enough since they do not remove the O(N) array assignment. This patch adds a preprocessing step - `xdl_trim_files` that runs before other preprocessing steps. It counts common prefix and suffix and lines in them (needed for displaying line number), without doing anything else. Testing with a crafted large (169MB) file, with minor change: ``` open('a','w').write(''.join('%s\n' % (i % 100000) for i in xrange(30000000) if i != 6000000)) open('b','w').write(''.join('%s\n' % (i % 100000) for i in xrange(30000000) if i != 6003000)) ``` Running xdiff by a simple binary [3], this patch improves the xdiff perf by more than 10x for the above case: ``` # xdiff before this patch 2.41s user 1.13s system 98% cpu 3.592 total # xdiff after this patch 0.14s user 0.16s system 98% cpu 0.309 total # gnu diffutils 0.12s user 0.15s system 98% cpu 0.272 total # (best of 20 runs) ``` It's still slightly slower than GNU diffutils. But it's pretty close now. Testing with real repo data: For the whole repo, this patch makes xdiff 25% faster: ``` # hg perfbdiff --count 100 --alldata -c d334afc585e2 --blocks [--xdiff] # xdiff, after ! wall 0.058861 comb 0.050000 user 0.050000 sys 0.000000 (best of 100) # xdiff, before ! wall 0.077816 comb 0.080000 user 0.080000 sys 0.000000 (best of 91) # bdiff ! wall 0.117473 comb 0.120000 user 0.120000 sys 0.000000 (best of 67) ``` For files that are long (ex. commands.py), the speedup is more than 3x, very significant: ``` # hg perfbdiff --count 3000 --blocks commands.py.i 1 [--xdiff] # xdiff, after ! wall 0.690583 comb 0.690000 user 0.690000 sys 0.000000 (best of 12) # xdiff, before ! wall 2.240361 comb 2.210000 user 2.210000 sys 0.000000 (best of 4) # bdiff ! wall 2.469852 comb 2.440000 user 2.440000 sys 0.000000 (best of 4) ``` [1]: https://phab.mercurial-scm.org/D2631 [2]: https://phab.mercurial-scm.org/D2634 [3]: ``` // Code to run xdiff from command line. No proper error handling. #include <stdlib.h> #include <unistd.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #include "mercurial/thirdparty/xdiff/xdiff.h" #define ensure(x) if (!(x)) exit(255); mmfile_t readfile(const char *path) { struct stat st; int fd = open(path, O_RDONLY); fstat(fd, &st); mmfile_t file = { malloc(st.st_size), st.st_size }; ensure(read(fd, file.ptr, st.st_size) == st.st_size); close(fd); return file; } int main(int argc, char const *argv[]) { mmfile_t a = readfile(argv[1]), b = readfile(argv[2]); xpparam_t xpp = {0}; xdemitconf_t xecfg = {0}; xdemitcb_t ecb = {0}; xdl_diff(&a, &b, &xpp, &xecfg, &ecb); return 0; } ``` Differential Revision: https://phab.mercurial-scm.org/D2686

transaction: add a name and a __repr__ implementation (API) This has been useful for me for debugging. Differential Revision: https://phab.mercurial-scm.org/D2758

phabricator: update doc string for deprecated token argument Differential Revision: https://phab.mercurial-scm.org/D2755

phabricator: print deprecation warning only once Differential Revision: https://phab.mercurial-scm.org/D2754

tests: add a few tests involving --collapse and rebase.singletransaction=1 I'm about to change the rebase code quite a bit and this was poorly tested. Differential Revision: https://phab.mercurial-scm.org/D2757

tests: simplify test-rebase-transaction.t The file was extracted from test-rebase-base.t in 8cef8f7d51d0 (test-rebase-base: clarify it is about the "--base" flag, 2017-10-05). This patch follows up that and clarifies the new file's purpose and simplifies it a bit. Differential Revision: https://phab.mercurial-scm.org/D2756

hgweb: parse and store HTTP request headers WSGI transmits HTTP request headers as HTTP_* environment variables. We teach our parser about these and hook up a dict-like data structure that supports case insensitive header manipulation. Differential Revision: https://phab.mercurial-scm.org/D2742

wireprotoserver: remove broken optimization for non-httplib client There was an experimental non-httplib client in core for several years. It was removed a week or so ago. We kept the optimization for this client in the server code. I'm not sure if that was intended or not. But it doesn't matter: the code was wrong. Because the code was accessing a WSGI environment dict, it needed to access the HTTP_X_HGHTTP2 key to actually read the HTTP header. So the code deleted by this commit wasn't actually doing anything meaningful. Doh. Differential Revision: https://phab.mercurial-scm.org/D2741

wireprotoserver: move all wire protocol handling logic out of hgweb Previous patches from several days ago worked to isolate processing of HTTP wire protocol requests to wireprotoserver. We still had a little logic in hgweb. If feels like the right time to finish the job. This commit moves WSGI request servicing from hgweb to wireprotoserver. The ugly dict holding the parsed request is no more. I think the new code is cleaner. As part of this, we now process wire protocol requests before the block to obtain the "query" variable. This makes it clear that this wonky "query" variable is not used by the wire protocol. The wonkiest part about this code is the HTTP 404. I'm actually not sure what all is going on here. It looks like the code is trying to prevent URL with path components that specify a command from not working. That part I grok. What I don't grok is why we need to send a 404. I would think it would be OK to no-op and let another handler try to service the request. But if we do this, we get some subrepo test failures. So it looks like something is expecting the HTTP 404 and reacting to it in a specific way. It /might/ be possible to change the behavior here. But it isn't something I'm comfortable doing because I don't understand the problem space. Differential Revision: https://phab.mercurial-scm.org/D2740

hgweb: use parsed request to construct query parameters The way hgweb routes requests is kind of bonkers. If PATH_INFO is set, we take the URL path after the repository. Otherwise, we take the first part of the query string before "&" and the part before ";" in that. We then kinda/sorta treat this as a path and route based on that. This commit ports that code to use the parsed request object. This required a new attribute on the parsed request to indicate whether there is any PATH_INFO. The new code still feels a bit convoluted for my liking. But we'll need to rewrite more of the code before a better solution becomes apparant. This code feels strictly better since we're no longer doing low-level WSGI manipulation during routing. Differential Revision: https://phab.mercurial-scm.org/D2739

hgweb: only recognize wire protocol commands from query string (BC) Previously, we attempted to parse the wire protocol command from `req.form`. Data could have come from the query string or POST form data. The wire protocol states that the command must be declared in the query string. And AFAICT all Mercurial releases from at least 1.0 send the command in the query string. So let's actual require this behavior. This is technically BC. But I'm not sure how anyone in the wild would encounter this. POST has historically been used for sending bundle data. So there's no opportunity to encode arguments there. And the experimental HTTP POST args also takes over the body. So the only way someone would be impacted by this is if they wrote a custom client that both used POST for everything and sent arguments via the HTTP body. I don't believe such a client exists. .. bc:: The HTTP wire protocol server no longer accepts the ``cmd`` argument to control which command to run via HTTP POST bodies. The ``cmd`` argument must be specified on the URL query string. Differential Revision: https://phab.mercurial-scm.org/D2738

hgweb: teach WSGI parser about query strings Currently, req.form uses cgi.parse() to populate form data. Depending on the request, form data can come from POST multipart/form-data, application/x-www-form-urlencoded, or the URL query string. Putting all these things into one data structure makes it difficult to reason about how exactly parameters got to the request. It can lead to wonkiness such as pulling parameters from both the URL and POST data. This commit teaches our WSGI request parser about argument data in query strings. We populate fields containing the query string data and only the query string data so it can't be confused with POST data. Differential Revision: https://phab.mercurial-scm.org/D2737

hgweb: use the parsed application path directly Previously, we assigned a custom system string with a trailing slash to wsgirequest.url. The addition of the trailing slash felt arbitrary and seems to go against how things typically work in WSGI. We also want our URLs to be bytes, not system strings. And, assigning a custom attribute to wsgirequest felt wrong. This commit fixes all those things by removing the trailing slash from the app path, changing consumers to use that variable and to use it without a trailing slash, and removing the custom attribute from wsgirequest. We preserve the trailing slash on {url}. Also, makebreadcrumb strips the trailing slash. So no change to it was needed. Differential Revision: https://phab.mercurial-scm.org/D2736

hgweb: use computed base URL from parsed request Let's not reinvent URL construction in a function that runs the templater. Differential Revision: https://phab.mercurial-scm.org/D2735

hgweb: parse WSGI request into a data structure Currently, our WSGI applications (hgweb_mod and hgwebdir_mod) process the raw WSGI request instance themselves. This means they have to talk in terms of system strings. And they need to know details about what's in the WSGI request. And in the case of hgweb_mod, it is doing some very funky things with URL parsing to impact dispatching. The code is difficult to read and maintain. This commit introduces parsing of the WSGI request into a higher-level and easier-to-reason-about data structure. To prove it works, we hook it up to hgweb_mod and use it for populating the relative URL on the request instance. We hold off on using it in more places because the logic in hgweb_mod is crazy and I don't want to involve those changes with review of the parsing code. The URL construction code has variations that use the HTTP: Host header (the canonical WSGI way of reconstructing the URL) and with the use of SERVER_NAME. We need to differentiate because hgweb is currently using SERVER_NAME for URL construction. Differential Revision: https://phab.mercurial-scm.org/D2734

hgweb: always use "?" when writing session vars This code resolves a string to insert in URLs as part of a query string. Essentially, it resolves the {sessionvars} template keyword, which is used by hgweb templates to build a URL as a string. The whole approach here feels wrong because there's no way of knowing when this code runs how the final URL will look. There could be additional URL fragments added before this template keyword that add a query string component. Furthermore, I don't think there's *any* for req.url to have a query string. That's because the code that populates this variable only takes SCRIPT_NAME and REPO_NAME into account. The "?" character it is searching for would only be added if some code attempted to add QUERY_STRING to the URL. Hacking the code up to raise if "?" is present in the URL yields a clean test suite run. I'm not sure if we broke this code or if it has always been broken. Anyway, this commit removes support for emitting "&" as the first character in {sessionvars} and makes it always emit "?", which is what it was always doing before AFAICT. Differential Revision: https://phab.mercurial-scm.org/D2733

hgweb: rename req to wsgireq We will soon introduce a parsed WSGI request object so we don't have to concern ourselves with low-level WSGI matters. Prepare for multiple request objects by renaming the existing one so it is clear it deals with WSGI. We also remove a symbol import to avoid even more naming confusion. # no-check-commit because of some new foo_bar naming that's required Differential Revision: https://phab.mercurial-scm.org/D2732

hgweb: validate WSGI environment dict The wsgiref.validate module contains useful functions for validating that various WSGI data structures are proper. This commit adds validation of the environment dict to our built-in HTTP server, which turns an HTTP request into an environment dict. The check discovered that we weren't always setting QUERY_STRING, which would cause the cgi module to fall back to sys.argv. So we change things to always set QUERY_STRING. The check passes on Python 2 and 3. Differential Revision: https://phab.mercurial-scm.org/D2731

hgweb: ensure all wsgi environment values are str Previously, we had a few entries that were bytes on Python 3. PEP-0333 states that all entries must be the native str type (bytes on Python 2, str on Python 3). This required a number of changes to hgweb_mod to unbreak things on Python 3. I suspect there still may be some regressions. I'm going to introduce a data structure that represents a parsed WSGI request in upcoming commits. This will hold bytes and will allow us to stop using raw literals throughout the WSGI code. Differential Revision: https://phab.mercurial-scm.org/D2730

wireproto: formalize permissions checking as part of protocol interface Per the inline comment desiring to formalize permissions checking in the protocol interface, we do that. I'm not convinced this is the best way to go about things. I would love for there to e.g. be a better exception for denoting permissions problems. But it does feel strictly better than snipping attributes on the proto instance. Differential Revision: https://phab.mercurial-scm.org/D2719

wireproto: declare permissions requirements in @wireprotocommand (API) With the security patches from 4.5.2 merged into default, we now have a per-command attribute defining what permissions are needed to run that command. We now have a richer @wireprotocommand that can be extended to record additional command metadata. So we port the permissions mechanism to be based on @wireprotocommand. .. api:: hgweb_mod.perms and wireproto.permissions have been removed. Wire protocol commands should declare their required permissions in the @wireprotocommand decorator. Differential Revision: https://phab.mercurial-scm.org/D2718

wireprotoserver: check permissions in main dispatch function The permissions checking code merged from stable is out of place in the refactored hgweb_mod module. This commit moves the main call to wireprotoserver. We still have some lingering code in hgweb_mod. This will get addressed later. Differential Revision: https://phab.mercurial-scm.org/D2717

wireprotoserver: check if command available before calling it The previous behavior was just plain wrong. I have no clue how it landed. My guess is a merge conflict resolution gone wrong on my end a few weeks ago. Differential Revision: https://phab.mercurial-scm.org/D2716

py3: drop encoding.strio() Its buffered nature makes TextIOWrapper unsuitable for temporarily wrapping bytes I/O.

ui: adjust Windows workaround to new _readline() code It's only needed when rawinput() is called. Also made it Py3 compatible.

ui: do not use rawinput() when we have to replace sys.stdin/stdout See the inline comment for why. The current Python3 hack doesn't work if more than one user inputs are expected because TextIOWrapper fills its internal buffer at the first read() request. Maybe we could write an unbuffered TextIOWrapper, but I don't want to make things more complicated. Instead, this patch reinvents raw_input(' ') of no readline support.

ui: do not try readline support if fin/fout aren't standard streams It's unlikely for a non-stdio stream to be a tty. Minimizing readline support makes it much simpler to work around the unicode input() function of Python 3. This also works on chg which duplicates client's tty to stdio fds.

util: add public isstdin/isstdout() functions

ui: add debug commands to test interactive prompt Interactive operations aren't easily covered by tests. So let's add commands to test them manually.

ui: inline util.bytesinput() into ui._readline() Prepares for rework of Python 3 support, which is currently broken due to read-ahead buffer of TextIOWrapper.

hgk: stop using util.bytesinput() to read a single line from stdin Appears that the stdio here is an IPC channel between hg and hgk (tk) processes, which shouldn't need a fancy readline support.

bookmarks: test for exchanging long bookmark names (issue5165) As far as I can tell a test for a long bookmark name never actually got added when this was fixed. Let's document that a 300 byte bookmark is something we're supporting (this patch started out life over a year ago as a way for me to validate the problem, and I recently found it.) I think the nonzero exits from the push operations are only because no new changesets get exchanged. Please correct me if I'm wrong on that. :) Differential Revision: https://phab.mercurial-scm.org/D2727

phabricator: follow-up phab auth improvements with backwards compat mode We'll rip this out before we ship 4.6, but this gives people a window to migrate. Differential Revision: https://phab.mercurial-scm.org/D2703

phabricator: specify API tokens per host, rather than per repo Differential Revision: https://phab.mercurial-scm.org/D1919

py3: drop b'' from generate-working-copy-states.py output I could make everything bytes, but decoding filename seemed easier and we don't have to worry about non-ascii filenames here.

py3: make test-commit-multiple.t byte-safe

py3: fix type of default username

py3: read/write plain lock file in binary mode A lock file shouldn't contain '\n', so this isn't a BC.

util: stop calling os.stat_float_times() It had Python-wide side effects, and it disappears in 3.7.0. As of this change, we're mostly working on 3.7.0b2. There are a few worrying failures, mostly around regular expressions, but we'll have to tackle those separately. Differential Revision: https://phab.mercurial-scm.org/D2697

cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime The latter is floating point by default, and we've been doing os.stat_float_times(False). Unfortunately, os.stat_float_times was removed between Python 3.7.0a1 and 3.7.0b2, so we have to stop using it. Differential Revision: https://phab.mercurial-scm.org/D2696

osutil: implement minimal __getitem__ compatibility on our custom listdir type We previously declined to do this, but the removal of the deprecated os.stat_float_times() method in Python 3.7 forces our hand. Differential Revision: https://phab.mercurial-scm.org/D2695

hgweb: adapt to socket._fileobject changes in Python 3 Differential Revision: https://phab.mercurial-scm.org/D2688

debugcommands: fix some %r output with bytestr() wrappers Almost fixes test-merge-tools.t. I think the remaining failure there is due to some overspecified tempfile names. Differential Revision: https://phab.mercurial-scm.org/D2675

tests: add test for issue 5494 but with --collapse This was not fixed, so the test case currently demonstrates the breakage. Differential Revision: https://phab.mercurial-scm.org/D2714

tests: .hg/merge is a directory, so use `test -d` This part of test-rebase-interrupts.t would have passed before the fix in a580b2d65ded (rebase: make sure merge state is cleaned up for no-op rebases (issue5494), 2017-05-18). Differential Revision: https://phab.mercurial-scm.org/D2713

rebase: only store collapse message once The message is determined by the user passing --message or --log when the rebase is started. There's no need to write it to a file for each rebased commit; writing it once at the start of the rebase is enough. Differential Revision: https://phab.mercurial-scm.org/D2712

rebase: collapse two nested if-conditions Also change the order since it feel to me like it's more about --collapse than it is about --keep. Differential Revision: https://phab.mercurial-scm.org/D2711

rebase: reduce scope of "dsguard" variables a bit Differential Revision: https://phab.mercurial-scm.org/D2710

rebase: remove unused argument "state" from rebasenode() Differential Revision: https://phab.mercurial-scm.org/D2709

rebase: delete obsolete internal "keepopen" option The option was apparently introduced for use by the "pbranch" extension, see f2558a8228be (rebase: add option to not commit after a collapsing, 2010-02-07). However, it doesn't seem like it was ever used by that extension (according to `hg grep` in a clone of https://bitbucket.org/parren/hg-pbranch/), so let's delete it. Differential Revision: https://phab.mercurial-scm.org/D2708

releasenotes: allow notes for multiple directives in a single changeset This problem was caught in da91e7309daf8ffc51bf3e6f4b2d8a16ef5af95a. This patch just makes sure there is no warning when we encounter such a case. Differential Revision: https://phab.mercurial-scm.org/D2254

releasenotes: mention changeset with warning and abort Output the changeset hash with the warning/abort message just to know where things messed up. Differential Revision: https://phab.mercurial-scm.org/D2253

releasenotes: replace abort with warning while parsing (issue5775) During the 4.5 development cycle, the extension broke on two different changesets. This fixes the issue by ensuring that it just throws a warning when it encounters unexpected behaviour, instead of aborting. Differential Revision: https://phab.mercurial-scm.org/D2255

archival: fileit should not use atomictemp, causes performance regression Differential Revision: https://phab.mercurial-scm.org/D2704

perf: teach perfbdiff to call blocks() and to use xdiff Differential Revision: https://phab.mercurial-scm.org/D2624

fuzz: fix xdiff build Recent xdiff code cleanups removed some files and changed some structures. Update fuzz code. Differential Revision: https://phab.mercurial-scm.org/D2707

xdiff: remove xmerge related logic hg has its own merge algorithm with flexible config options. Differential Revision: https://phab.mercurial-scm.org/D2706

xdiff: remove xemit related logic xemit handles "diff formatting and output" with options like context lines, whether show function names, etc. That is handled more cleanly at a higher level in hg. Removing context line parameters would also make the trimming logic (D2686) cleaner and more confident. See [1]. [1]: https://github.com/git/git/commit/d2f82950a9226ae1102a7a97f03440a4bf8c6c09 Differential Revision: https://phab.mercurial-scm.org/D2705

xdiff: remove unused structure, functions, and constants `bdiffparam_t` is unused. `xdl_fall_back_diff` is no longer used after D2573. `XDL_MMB_READONLY`, `XDL_MMF_ATOMIC` are unused. `XDL_BDOP*` are unused since there is no xdiff binary diff algorithm. `anchors` feature is not used. It's also relatively new in git. Differential Revision: https://phab.mercurial-scm.org/D2684

xdiff: remove whitespace related feature In Mercurial, whitespace related handling are done at a higher level than the low-level diff algorithm so "ignore spaces". So it's not used by mdiff. Some of the upcoming optimizations would be more difficult with whitespace related features kept. So let's remove them. Differential Revision: https://phab.mercurial-scm.org/D2683

merge with stable There were a handful of merge conflicts in the wire protocol code due to significant refactoring in default. When resolving the conflicts, I tried to produce the minimal number of changes to make the incoming security patches work with the new code. I will send some follow-up commits to get the security patches better integrated into default.

sslutil: some more forcebytes() on some exception messages At this point, test-https.t no longer dumps tracebacks everywhere. Instead, we get some results that look like we're not adequately finding things in hg's configuration, which should be manageable (if somewhat annoying to find and fix.) Differential Revision: https://phab.mercurial-scm.org/D2690

sslutil: sslcontext needs the cipher name as a sysstr Differential Revision: https://phab.mercurial-scm.org/D2689

sslutil: lots of unicode/bytes cleanup In general, we handle hostnames as bytes, except where Python forces them to be unicodes. This fixes all the tracebacks I was seeing in test-https.t, but there's still some ECONNRESET weirdness that I can't hunt down... Differential Revision: https://phab.mercurial-scm.org/D2687

debugwireproto: handle unimplemented util.poll() for Windows This is the same logic used in sshpeer.doublepipe. It doesn't completely fix test-ssh-proto{,-unbundle}.t ("read(-1) -> X" is changed to "read(X) -> X", the order of some lines are changed, and abort messages seem to be missing), but it cuts down a ton on the failure spew.

py3: byte-stringify test-blackbox.t

py3: byte-stringify blackbox-readonly-dispatch.py # skip-blame because just adding some b''

py3: make blackbox-readonly-dispatch.py use ui instead of print()

py3: fix int formatting of "incoming changes" log

largefiles: use %d instead of %s to process ints Differential Revision: https://phab.mercurial-scm.org/D2677

transaction: fix an error string with bytestr() on a repr()d value Fixes test-rollback.t on Python 3. Differential Revision: https://phab.mercurial-scm.org/D2674

py3: work around comparison between int and None in tagmerge

py3: do not mutate dict while iterating in tagmerge

py3: fix type of ui.configitems(ignoresub=True) result

py3: don't use str() to stringify pushloc

py3: byte-stringify test-config.t and test-config-env.py

py3: use startswith() instead of slicing to detect leading whitespace

archival: use py3 friendly replacements for chr() and long() Differential Revision: https://phab.mercurial-scm.org/D2673

archival: ensure file mode for gzipfile is sysstr Differential Revision: https://phab.mercurial-scm.org/D2672

archival: fix a missing r'' on a kwargs check # skip-blame just an r prefix Differential Revision: https://phab.mercurial-scm.org/D2671

py3: more passing tests (ten this time) Differential Revision: https://phab.mercurial-scm.org/D2670

util: fix unsafe url abort with bytestr() on url Differential Revision: https://phab.mercurial-scm.org/D2669

xdiff: fix builds on Windows This works on my ancient Fedora system too, without warnings. There are, however, warnings about various 64 to 32 bit conversions on Windows that need to be examined.