sshpeer: make pipe polling code more explicit "hasbuffer" is a property on our special bufferedinputpipe class. When reading this code, I thought it might have had something special to do properties on built-in types. But "hasbuffer" doesn't appear in the CPython code base for either 2.7 or 3.7, so the answer is no. Let's make the code more explicit about the fact that it deals with our special bufferedinputpipe type. Differential Revision: https://phab.mercurial-scm.org/D2382

tests: store protocol payload in files Upcoming changes to version 2 of the SSH protocol will introduce binary components to the protocol. It will be easier to eliminate trailing newlines and use binary in the tests if the protocol payload is being generated by Python. So use inline Python to write payloads to files and pipe those files to server processes instead of shell strings/variables. Differential Revision: https://phab.mercurial-scm.org/D2381

sshpeer: return framed file object when needed Currently, wireproto.wirepeer has a default implementation of _submitbatch() and sshv1peer has a very similar implementation. The main difference is that sshv1peer is aware of the total amount of bytes it can read whereas the default implementation reads the stream until no more data is returned. The default implementation works for HTTP, since there is a known end to HTTP responses (either Content-Length or 0 sized chunk). This commit teaches sshv1peer to use our just-introduced "cappedreader" class for wrapping a file object to limit the number of bytes that can be read. We do this by introducing an argument to specify whether the response is framed. If set, we returned a cappedreader instance instead of the raw pipe. _call() always has framed responses. So we set this argument unconditionally and then .read() the entirety of the result. Strictly speaking, we don't need to use cappedreader in this case and can inline frame decoding/read logic. But I like when things are consistent. The overhead should be negligible. _callstream() and _callcompressable() are special: whether framing is used depends on the specific command. So, we define a set of commands that have framed response. It currently only contains "batch." As a result of this change, the one-off implementation of _submitbatch() in sshv1peer can be removed since it is now safe to .read() the response's file object until end of stream. cappedreader takes care of not overrunning the frame. Differential Revision: https://phab.mercurial-scm.org/D2380

sshpeer: move logic for sending a request into a new function The **args being used to pass arbitrary command arguments is limiting because it makes it harder to control behavior of the function. We factor most of _callstream() into a new function that doesn't use **args. Differential Revision: https://phab.mercurial-scm.org/D2379

sshpeer: rename _recv and _send to _readframed and _writeframed Because it is reading and writing a chunk of data with a well-defined size. "recv" and "send" make it sound like things are a direct proxy to the underlying pipe, which they aren't. Differential Revision: https://phab.mercurial-scm.org/D2378

util: add a file object proxy that can read at most N bytes Sometimes we have data of a known size within a stream. For performance reasons, we don't want to pre-read this data (we want to allow consumers to read on demand). For simplicitly reasons, we don't want callers to necessarily know their data is coming from within an outer stream and there is a limit to how much they should read. The class introduced by this commit provides a very simple proxy around an underlying file object that allows the consumer to .read() up to N bytes from the file object. Attempts to read past this many bytes results in a simulated EOF. Differential Revision: https://phab.mercurial-scm.org/D2377

patches: release the GIL while applying the patch This will allow multiple threads to apply patches at the same time.

perfbranchmap: allow to select the filter to benchmark Running the branchmap computation on all filter levels can be expensive. Narrowing the run to some specific filters can speed up benchmarking time when working only on a subset of filter levels.

perfbranchmap: display 'unfiltered' for unfiltered performance This is slightly clearer than "None" and will help with coming changes to select the filter level we want timing for.

py3: two more narrow tests passing Differential Revision: https://phab.mercurial-scm.org/D2390

narrowbundle2: more kwargs native string fixes This gets test-narrow.t to *almost* pass. Something appears to be borked in producing bundles, but only some of the time? I'm lost, but this change is at least a clear improvement. # skip-blame just more r prefixes on strings Differential Revision: https://phab.mercurial-scm.org/D2389

py3: whitelist another 11 passing tests This is most of narrow. There's still some buglets at the margins, but it's pretty good progress for not a lot of work. Differential Revision: https://phab.mercurial-scm.org/D2388

narrowbundle2: use native string to get kwargs from dict # skip-blame just some r prefixes on strings Differential Revision: https://phab.mercurial-scm.org/D2387

narrowbundle2: drop legacy getcgkwargs variable I think this was around as part of support for some older hg internals. It's not needed any more. Differential Revision: https://phab.mercurial-scm.org/D2386

fancyopts: add support for custom multi-arg opts in fancyopts.py This allows for more complex multi-arg opt logic, such as "--sum 1 --sum 2" -> 3, "--csv alice,bob --csv charlie" -> ["alice","bob","charlie"]. The current support for callables is insufficient for this. This is done by introducing a 'customopt' class which can be extended for more powerful opts logic. All existing opt-types are converted to use this class, simplifying the fancyopts() logic. Differential Revision: https://phab.mercurial-scm.org/D2090

narrowcommands: add some missing strkwargs calls for py3 # skip-blame because it's just r prefixes Differential Revision: https://phab.mercurial-scm.org/D2367

narrowwirepeer: add some strkwargs to fix a crash on py3 # skip-blame because it's just some r prefixes Differential Revision: https://phab.mercurial-scm.org/D2366

narrowchangegroup: remove backwards compatibility with old hg This was missed in the initial import of narrowhg, but was detected by the Python 3 porting effort once I got enough other things in narrow fixed. Differential Revision: https://phab.mercurial-scm.org/D2370

narrowbundle2: replace map() with equivalent list comprehension The result of this gets used as a list in core code, so the generator returned by map() on Python 3 is a problem. Differential Revision: https://phab.mercurial-scm.org/D2369

narrowbundle2: this dict contains native strings, look kws up as such We could also do a byteskwargs dance, but that seems silly given that we only need this one element. Differential Revision: https://phab.mercurial-scm.org/D2368

tests: port extension in test-narrow-expanddirstate.t to Python 3 Differential Revision: https://phab.mercurial-scm.org/D2365

py3: use list comprehensions instead of filter where we need to eagerly filter These two uses of filter() are then checked for truthiness, but on Python 3: >>> bool(filter(None, [])) True So we need to stop depending on that. Fortunately it's easy to replace the filter with an equivalent list comprehension. Differential Revision: https://phab.mercurial-scm.org/D2364

narrow: use list comprehension instead of filter for filtering lists filter() returns a generator on Python 3, which causes these filters to break things. Differential Revision: https://phab.mercurial-scm.org/D2363

py3: whitelist another eight passing tests Differential Revision: https://phab.mercurial-scm.org/D2362

mq: don't reimplement any() Differential Revision: https://phab.mercurial-scm.org/D2376

mq: don't reimplement any() Differential Revision: https://phab.mercurial-scm.org/D2375

strip: don't reimplement any() Differential Revision: https://phab.mercurial-scm.org/D2374

convert: don't reimplement any() Differential Revision: https://phab.mercurial-scm.org/D2373

verify: don't reimplement any() Differential Revision: https://phab.mercurial-scm.org/D2372

walkrepos: don't reimplement any() Differential Revision: https://phab.mercurial-scm.org/D2371

py3: make sure we open file in bytes mode Differential Revision: https://phab.mercurial-scm.org/D2360

py3: add b'' to test-ui-color.py # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D2359

debuginstall: strip double quotes from editorbin on Windows The unconditional posix style shlex.split() prior to 94a1ff16f362 handled this. This isn't mutually exclusive with stripping the quotes in util.findexe()- if the editor can't be found, this command prints out the string, inside single quotes.

pycompat: correct the shlex.split() proxy method signature in py3

merge with stable

convert: don't use type as a variable name Differential Revision: https://phab.mercurial-scm.org/D2358

convert: don't use bytes as a variable name Differential Revision: https://phab.mercurial-scm.org/D2357

py3: add a r'' prefix to prevent transformer from adding b'' # skip-blame as only r'' prefix was added Differential Revision: https://phab.mercurial-scm.org/D2356

py3: use pycompat.byteskwargs in hgext/convert/ Differential Revision: https://phab.mercurial-scm.org/D2355

py3: add b'' prefixes in test-mdiff.py # skip-blame because we are just adding b'' Differential Revision: https://phab.mercurial-scm.org/D2354

py3: use dict.items() instead of dict.iteritems() in tests dict.iteritems() is not present in Python 3. Differential Revision: https://phab.mercurial-scm.org/D2353

py3: add b'' prefixes in test-transplant.t # skip-blame because we are just adding b'' Differential Revision: https://phab.mercurial-scm.org/D2352

py3: add b'' prefixes in fakepatchtime.py # skip-blame because we are just adding b'' Differential Revision: https://phab.mercurial-scm.org/D2351

py3: add b'' prefixes in fakedirstatewritetime.py # skip-blame because we are just adding b'' Differential Revision: https://phab.mercurial-scm.org/D2350

py3: use '%d' to convert integer to bytes Differential Revision: https://phab.mercurial-scm.org/D2349

hgweb: show each obsfateentry on its own line Commits with more than one reason for being obsolete used to just show obsfate entries all on one line, and that doesn't look good. Let's show each entry on its own line. In paper and coal the lines are simply split using a <br> element, and in other hgweb themes each entry has its own table header. This is done by analogy with changeset parents and children -- in paper and coal they are all put into one table row, and everywhere else each one gets a separate row.

tests: check obsolete changeset with two obsfate entries It's more interesting to see how a changeset that was both rewritten and split looks in hgweb.

py3: whitelist another six passing tests Differential Revision: https://phab.mercurial-scm.org/D2348

lock: delay is numeric, use %d for formatting Differential Revision: https://phab.mercurial-scm.org/D2347

debugbuilddag: use '%d' instead of str() to get numbered lines Differential Revision: https://phab.mercurial-scm.org/D2346

tests: add missing b prefixes in test-pending.t # skip-blame more b prefixes Differential Revision: https://phab.mercurial-scm.org/D2345

merge: make a copy of dict.items() before mutating the dict during iteration Differential Revision: https://phab.mercurial-scm.org/D2344

largefiles: give some **opts some strkwargs love Differential Revision: https://phab.mercurial-scm.org/D2343

wireproto: fix lingering str(exception) with util.forcebytestr(exception) Differential Revision: https://phab.mercurial-scm.org/D2342

scmutil: fix requires-file isalnum() check on first byte Slice instead of subscript to get a bytes from a bytes. # skip-blame just a py3 slice-instead-of-subscript change Differential Revision: https://phab.mercurial-scm.org/D2341

largefiles: mark headre as bytes regex Differential Revision: https://phab.mercurial-scm.org/D2340

largfiles: replace filter() with listcomp when result needs to be a list filter() is a generator on Python 3, but these cases are used as lists. Differential Revision: https://phab.mercurial-scm.org/D2339

largefiles: make scheme regex a bytes regex # skip-blame just a b prefix Differential Revision: https://phab.mercurial-scm.org/D2338

debuginstall: do not parse editor command in posix way on Windows An editor command is executed by a system shell, which is cmd.exe on Windows.

py3: use bytes() to byte-stringify url object in url.py Perhaps we'll have to convert it back and forth from/to bytes and unicode at urllib boundary, but at least util.hidepassword() wants a byte string.

url: show full url of proxy server in debug message This only fixes the debug message spotted by issue5796.

py3: make sure we are doing integer division by using '//' Differential Revision: https://phab.mercurial-scm.org/D2337

py3: make sure regex is bytes in crecord.py # skip-blame because we are just adding b'' Differential Revision: https://phab.mercurial-scm.org/D2336

py3: use "%d" for integers instead of "%s" Differential Revision: https://phab.mercurial-scm.org/D2335

py3: make sure we open the files in bytes mode Differential Revision: https://phab.mercurial-scm.org/D2333

py3: whitelist another three passing tests Differential Revision: https://phab.mercurial-scm.org/D2318

manifest: use list(dict) instead of dict.keys() to get a list of keys Differential Revision: https://phab.mercurial-scm.org/D2317

manifest: correct the one use of iterkeys() on a dict As far as I can tell, this is the only iterkeys() in this file that was actually on a dict. The rest are on custom manifest types. Differential Revision: https://phab.mercurial-scm.org/D2316

py3: use default dict iterator instead of iterkeys These are the easy cases. Some cases in manifest.py will require more careful inspection. Differential Revision: https://phab.mercurial-scm.org/D2315

py3: whitelist five more passing tests Differential Revision: https://phab.mercurial-scm.org/D2314

httppeer: headers are native strings # skip-blame just marking some native strings Differential Revision: https://phab.mercurial-scm.org/D2313

hgweb: header dict entries are native strings # skip-blame just marking some native strings Differential Revision: https://phab.mercurial-scm.org/D2312

keepalive: headers are native strings, mark them as such # skip-blame just marking some native strings Differential Revision: https://phab.mercurial-scm.org/D2311

py3: use range instead of xrange on py3 in tests/test-ui-verbosity.py xrange is not present on Python 3. Differential Revision: https://phab.mercurial-scm.org/D2332

py3: use range instead of xrange in tests/test-revset2.t xrange is not present on Python 3. Differential Revision: https://phab.mercurial-scm.org/D2331

py3: use range instead of xrange in tests/test-mq-qimport.t xrange is not present on Python 3. Differential Revision: https://phab.mercurial-scm.org/D2330

py3: use range instead of xrange in tests/test-issue4074.t xrange in not present on Python 3. Differential Revision: https://phab.mercurial-scm.org/D2329

py3: use range instead of xrange in tests/test-convert-mtn.t xrange in not present on Python 3. This change will make the test a bit slower on Python 2. Differential Revision: https://phab.mercurial-scm.org/D2328

py3: use range instead of xrange in tests/test-commandserver.t xrange is not present on Python 3. Differential Revision: https://phab.mercurial-scm.org/D2327

py3: use range instead of xrange in tests/test-walk.t xrange in not present in python 3. Differential Revision: https://phab.mercurial-scm.org/D2326

py3: use range instead of xrange in tests/test-mq-missingfiles.t xrange is not present in python 3. Differential Revision: https://phab.mercurial-scm.org/D2325

py3: use range instead of xrange on py3 in tests/test-filecache.py xrange is not present on py3. Differential Revision: https://phab.mercurial-scm.org/D2324

py3: use range instead on xrange on py3 in tests/test-atomictempfile.py xrange is not available on Python 3. Differential Revision: https://phab.mercurial-scm.org/D2323

py3: add b'' prefix in test-mq-qrefresh-replace-log-message.t # ski-blame as we just added b'' Differential Revision: https://phab.mercurial-scm.org/D2322

py3: add b'' prefixes in test-mq-eol.t # skip-blame because we are just adding b'' Differential Revision: https://phab.mercurial-scm.org/D2321

py3: explicitly convert result of dict.items() into list Differential Revision: https://phab.mercurial-scm.org/D2320

py3: use pycompat.byteskwargs() to convert opts keys to bytes Differential Revision: https://phab.mercurial-scm.org/D2319

wireprotoserver: py3 helpfully calls adds HTTP_ to CONTENT_LENGTH Just handle both with a membership check, preferring the HTTP_ namespaced version. Sigh. Differential Revision: https://phab.mercurial-scm.org/D2310

tests: port inline Python in test-http-branchmap.t to Python 3 Differential Revision: https://phab.mercurial-scm.org/D2309

hgweb_mod: partially undo 8afc25e7effc to fix py3 We now bytes-ify the entire request, so this is wrong. Differential Revision: https://phab.mercurial-scm.org/D2308

tags: don't feed both int and None to min() They're not comparable on Python 3, and we can just map None to 0. Differential Revision: https://phab.mercurial-scm.org/D2307

tests: fix get-with-headers.py on python3 when writing to stdout Differential Revision: https://phab.mercurial-scm.org/D2306

webcommands: use pycompat.strkwargs as needed Differential Revision: https://phab.mercurial-scm.org/D2305

webcommands: unpack contents of length-1 dict portably ((k,v),) = dict.items() was suggested by some friends, but I feel like that's maybe too clever? Differential Revision: https://phab.mercurial-scm.org/D2304

bundle2: use pycompat.strkwargs as needed Differential Revision: https://phab.mercurial-scm.org/D2303

tests: add b prefixes to test-rebase-dest.t # skip-blame just b prefixes Differential Revision: https://phab.mercurial-scm.org/D2302

rebase: sort roots by revision Previously we were sorting them by whatever implicit sort order we got between changectx instances. Sorting by rev at least makes some sense, so I chose that. Differential Revision: https://phab.mercurial-scm.org/D2301

httppeer: use %d to format int Differential Revision: https://phab.mercurial-scm.org/D2300

extensions: reject any unicode strings in tables before loading This allows us to test hg on Python 3 without disabling third-party extensions which could pollute cmdtable for example.

py3: make test-extdiff.t pass on Python 3

py3: make regexp patterns bytes in extdiff

py3: fix bytes-unicode dance while building docstring of extdiff

py3: factor out byterepr() which returns an asciified value on py3

remotenames: don't use the default value of logfmt for namespaces logfmt is the format which is used to format the log output for that namespace. This patch passes "remote {bookmark|branch}: %s" as the logfmt. Space is not added after bookmark and branch to make output consistent with other details. Still this is not the best output. We may need to wrap getlogcolumns() to change spacing in the in built columns to match the remotenames one. lognames are also deleted as they are superseded by logfmt. Differential Revision: https://phab.mercurial-scm.org/D2277

py3: whitelist test-pull-http.t There's a lot of work left to do on network-related stuff, but I at least got one more test passing. Differential Revision: https://phab.mercurial-scm.org/D2296

scmutil: bytes-ify IOErrors before wrapping them in abort message Differential Revision: https://phab.mercurial-scm.org/D2292

py3: port f to Python 3 This involved a lot of b'' literals, conversion from %s to %d, and using a hashing mechanism that returns bytes instead of str. Differential Revision: https://phab.mercurial-scm.org/D2299

py3: replace file() with open() in test-convert-git.t file() is not present in Python 3. It also makes sure we write bytes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D2133

bundle: updates the help text for hg bundle (issue5744) Differential Revision: https://phab.mercurial-scm.org/D2278

py3: get bytes-repr of network errors portably This resolves a lot of weird issues in Python 3 around error strings. Differential Revision: https://phab.mercurial-scm.org/D2295

hgweb: open server logs in binary mode This is consistent with when we're logging to stdout, so we don't have to do something annoyingly complicated in the logging infrastructure. Differential Revision: https://phab.mercurial-scm.org/D2294

tests: add some b prefixes in test-http-bundle1.t # skip-blame just some b prefixes Differential Revision: https://phab.mercurial-scm.org/D2293

hgweb: correctly bytes-ify status, not string-ify Differential Revision: https://phab.mercurial-scm.org/D2291

printenv: port to python3 Differential Revision: https://phab.mercurial-scm.org/D2290

py3: whitelist another eight passing tests Differential Revision: https://phab.mercurial-scm.org/D2281

remotenames: port partway to python3 by using collections.MutableMapping test-logexchange.t doesn't pass after this, but at least the remotenames extension can be imported. Differential Revision: https://phab.mercurial-scm.org/D2280

tests: avoid referring to pvec in demandimport test Nothing else currently uses pvec, so we may want to delete it (but Augie Fackler says he may want to use it). To enable deletion, this patch replaces it by the error module in the demandimport test (any module works). However, since the error module had already been loaded at this point in the test (via the util module), I moved it earlier in the test so it's still not loaded (although I'm not sure if that's even relevant to the test). Differential Revision: https://phab.mercurial-scm.org/D2287

help: use cmdutil.parsealiases() to resolve command name This seems slightly better than parsing '^command|name' string by using an ad-hoc pattern.

help: list deprecated and experimental extensions if --verbose This is common in the other help topics.

hgweb: translate Abort in help command to 404 error c3784e3c3e8d changed the exception type to be raised if unknowncmd=False.