sshpeer: defer pipe buffering and stderr sidechannel binding The doublepipe and bufferedinputpipe types facilitate polling multiple pipes without blocking and for automatically forwarding output from the SSH server's stderr pipe to the ui as "remote: " output. This all happens automatically and callers don't need to worry about reading from multiple pipes. An upcoming change to version 2 of the SSH wire protocol will eliminate the use of stderr and move side-channel output into the "main" pipe. The SSH wire protocol will use a pair of unidirectional pipes - just like the HTTP protocol. In this future world, the doublepipe primitive isn't necessary because the stderr pipe won't be used. To prepare for eventually not using doublepipe, we delay the construction of this primitive from immediately after connection establishment to inside construction of the peer instance. The handshake occurs between these two events. So we had to teach the handshake code to read from stderr so any stderr output from the server is still attended to early in the connection lifetime. Differential Revision: https://phab.mercurial-scm.org/D2383

sshpeer: make pipe polling code more explicit "hasbuffer" is a property on our special bufferedinputpipe class. When reading this code, I thought it might have had something special to do properties on built-in types. But "hasbuffer" doesn't appear in the CPython code base for either 2.7 or 3.7, so the answer is no. Let's make the code more explicit about the fact that it deals with our special bufferedinputpipe type. Differential Revision: https://phab.mercurial-scm.org/D2382

tests: store protocol payload in files Upcoming changes to version 2 of the SSH protocol will introduce binary components to the protocol. It will be easier to eliminate trailing newlines and use binary in the tests if the protocol payload is being generated by Python. So use inline Python to write payloads to files and pipe those files to server processes instead of shell strings/variables. Differential Revision: https://phab.mercurial-scm.org/D2381

sshpeer: return framed file object when needed Currently, wireproto.wirepeer has a default implementation of _submitbatch() and sshv1peer has a very similar implementation. The main difference is that sshv1peer is aware of the total amount of bytes it can read whereas the default implementation reads the stream until no more data is returned. The default implementation works for HTTP, since there is a known end to HTTP responses (either Content-Length or 0 sized chunk). This commit teaches sshv1peer to use our just-introduced "cappedreader" class for wrapping a file object to limit the number of bytes that can be read. We do this by introducing an argument to specify whether the response is framed. If set, we returned a cappedreader instance instead of the raw pipe. _call() always has framed responses. So we set this argument unconditionally and then .read() the entirety of the result. Strictly speaking, we don't need to use cappedreader in this case and can inline frame decoding/read logic. But I like when things are consistent. The overhead should be negligible. _callstream() and _callcompressable() are special: whether framing is used depends on the specific command. So, we define a set of commands that have framed response. It currently only contains "batch." As a result of this change, the one-off implementation of _submitbatch() in sshv1peer can be removed since it is now safe to .read() the response's file object until end of stream. cappedreader takes care of not overrunning the frame. Differential Revision: https://phab.mercurial-scm.org/D2380

sshpeer: move logic for sending a request into a new function The **args being used to pass arbitrary command arguments is limiting because it makes it harder to control behavior of the function. We factor most of _callstream() into a new function that doesn't use **args. Differential Revision: https://phab.mercurial-scm.org/D2379

sshpeer: rename _recv and _send to _readframed and _writeframed Because it is reading and writing a chunk of data with a well-defined size. "recv" and "send" make it sound like things are a direct proxy to the underlying pipe, which they aren't. Differential Revision: https://phab.mercurial-scm.org/D2378

util: add a file object proxy that can read at most N bytes Sometimes we have data of a known size within a stream. For performance reasons, we don't want to pre-read this data (we want to allow consumers to read on demand). For simplicitly reasons, we don't want callers to necessarily know their data is coming from within an outer stream and there is a limit to how much they should read. The class introduced by this commit provides a very simple proxy around an underlying file object that allows the consumer to .read() up to N bytes from the file object. Attempts to read past this many bytes results in a simulated EOF. Differential Revision: https://phab.mercurial-scm.org/D2377

patches: release the GIL while applying the patch This will allow multiple threads to apply patches at the same time.

perfbranchmap: allow to select the filter to benchmark Running the branchmap computation on all filter levels can be expensive. Narrowing the run to some specific filters can speed up benchmarking time when working only on a subset of filter levels.

perfbranchmap: display 'unfiltered' for unfiltered performance This is slightly clearer than "None" and will help with coming changes to select the filter level we want timing for.

py3: two more narrow tests passing Differential Revision: https://phab.mercurial-scm.org/D2390

narrowbundle2: more kwargs native string fixes This gets test-narrow.t to *almost* pass. Something appears to be borked in producing bundles, but only some of the time? I'm lost, but this change is at least a clear improvement. # skip-blame just more r prefixes on strings Differential Revision: https://phab.mercurial-scm.org/D2389

py3: whitelist another 11 passing tests This is most of narrow. There's still some buglets at the margins, but it's pretty good progress for not a lot of work. Differential Revision: https://phab.mercurial-scm.org/D2388

narrowbundle2: use native string to get kwargs from dict # skip-blame just some r prefixes on strings Differential Revision: https://phab.mercurial-scm.org/D2387

narrowbundle2: drop legacy getcgkwargs variable I think this was around as part of support for some older hg internals. It's not needed any more. Differential Revision: https://phab.mercurial-scm.org/D2386

fancyopts: add support for custom multi-arg opts in fancyopts.py This allows for more complex multi-arg opt logic, such as "--sum 1 --sum 2" -> 3, "--csv alice,bob --csv charlie" -> ["alice","bob","charlie"]. The current support for callables is insufficient for this. This is done by introducing a 'customopt' class which can be extended for more powerful opts logic. All existing opt-types are converted to use this class, simplifying the fancyopts() logic. Differential Revision: https://phab.mercurial-scm.org/D2090

narrowcommands: add some missing strkwargs calls for py3 # skip-blame because it's just r prefixes Differential Revision: https://phab.mercurial-scm.org/D2367

narrowwirepeer: add some strkwargs to fix a crash on py3 # skip-blame because it's just some r prefixes Differential Revision: https://phab.mercurial-scm.org/D2366

narrowchangegroup: remove backwards compatibility with old hg This was missed in the initial import of narrowhg, but was detected by the Python 3 porting effort once I got enough other things in narrow fixed. Differential Revision: https://phab.mercurial-scm.org/D2370

narrowbundle2: replace map() with equivalent list comprehension The result of this gets used as a list in core code, so the generator returned by map() on Python 3 is a problem. Differential Revision: https://phab.mercurial-scm.org/D2369

narrowbundle2: this dict contains native strings, look kws up as such We could also do a byteskwargs dance, but that seems silly given that we only need this one element. Differential Revision: https://phab.mercurial-scm.org/D2368

tests: port extension in test-narrow-expanddirstate.t to Python 3 Differential Revision: https://phab.mercurial-scm.org/D2365

py3: use list comprehensions instead of filter where we need to eagerly filter These two uses of filter() are then checked for truthiness, but on Python 3: >>> bool(filter(None, [])) True So we need to stop depending on that. Fortunately it's easy to replace the filter with an equivalent list comprehension. Differential Revision: https://phab.mercurial-scm.org/D2364

narrow: use list comprehension instead of filter for filtering lists filter() returns a generator on Python 3, which causes these filters to break things. Differential Revision: https://phab.mercurial-scm.org/D2363

py3: whitelist another eight passing tests Differential Revision: https://phab.mercurial-scm.org/D2362

mq: don't reimplement any() Differential Revision: https://phab.mercurial-scm.org/D2376

mq: don't reimplement any() Differential Revision: https://phab.mercurial-scm.org/D2375

strip: don't reimplement any() Differential Revision: https://phab.mercurial-scm.org/D2374