tests: port inline extension in test-http-bundle1.t to py3 # skip-blame just b prefixes Differential Revision: https://phab.mercurial-scm.org/D3366

tests: add all missing b prefixes in reactor tests Both of these tests now pass on Python 3. # skip-blame just b prefixes. So many b prefixes. Differential Revision: https://phab.mercurial-scm.org/D3369

stringutil: ast.literal_eval needs a unicode on py3 Fortunately for us, this is really only used internally, so we can be lazy about the encoding here. test-wireproto-framing.py now passes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D3368

scmutil: introduce shortesthexnodeidprefix() We have scmutil.resolvehexnodeidprefix() for resolving a prefix to a full nodeid, so it makes sense to have the inverse method next to it. For now it just delegates to changelog.shortest(), but it will soon also make sure it's called on the unfiltered repo, to match resolvehexnodeidprefix(). Note that the change in show.py also makes it so the conversion from revnum to nodeid is done on the filtered repo, but that should be inconsequential since the revs are all from the filtered repo anyway. Differential Revision: https://phab.mercurial-scm.org/D3370

scmutil: use resolvehexnodeidprefix() from revsymbol() I should have copied this from changectx.__init__ into in 35b34202dd3b (context: handle partial nodeids in revsymbol(), 2018-04-08). Differential Revision: https://phab.mercurial-scm.org/D3310

scmutil: rename resolvepartialhexnodeid() to resolvehexnodeidprefix() I'm going to wrap revlog.shortest() in scmutil. I plan on calling it shortesthexnodeidprefix(). resolvehexnodeidprefix() matches that better. Also, "prefix" carries more information than "partial". Differential Revision: https://phab.mercurial-scm.org/D3309

scmutil: document that isrevsymbol() raises on ambiguous node prefix Differential Revision: https://phab.mercurial-scm.org/D3308

context: set stack level for deprecation warning This patch makes the deprecation warning print the caller of repo.__getitem__. That's not quite correct, since there could also be other callers of changectx.__init__, but it seems like the most helpful stack level in practice. Differential Revision: https://phab.mercurial-scm.org/D3307

util: set correct stack level on deprecation warnings Without this patch, you'll get something like this: <path>/mercurial/util.py:3784: DeprecationWarning: 'util.hgexecutable' is deprecated, use 'utils.procutil.hgexecutable' (but on one line) Differential Revision: https://phab.mercurial-scm.org/D3331

revset: pass in lookup function instead of repo (API) And document that it's only for legacy lookup. If we have a repo, we're likely to do more things where that shouldn't be done.

revset: drop support for posttreebuilthook() (API) AFAIK, the only user was the directaccess extension, which is in core now.

py3: make values bytes before passing into server.runservice() The values of opts dict still needed to be converted to bytes. Differential Revision: https://phab.mercurial-scm.org/D3330

py3: two more passing tests Differential Revision: https://phab.mercurial-scm.org/D3338

localrepo: add some overlooked strkwargs love for py3 Differential Revision: https://phab.mercurial-scm.org/D3337

py3: paper over differences in future exception handling It looks like Python 3's futures library lacks set_exception_info entirely. We'll just give up and use set_exception in that case. # no-check-commit because the underbar naming is just saner here Differential Revision: https://phab.mercurial-scm.org/D3336

py3: add b'' prefixes to make values bytes # skip-blame beacuse just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3329

py3: use b"%d" instead of str() to convert int to bytes While I was here, I added 'and None' to suppress return values of .write() calls. Differential Revision: https://phab.mercurial-scm.org/D3328

py3: add b'' prefixes to make values bytes # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3327

py3: use str variables to check keys in request header The values in header are of str type. Differential Revision: https://phab.mercurial-scm.org/D3326

py3: make sure curses.tigetstr() first argument is a str Differential Revision: https://phab.mercurial-scm.org/D3325

py3: use stringutil.forcebytestr() instead of str() We need to convert errors to bytes using stringutil.forcebytestr() Differential Revision: https://phab.mercurial-scm.org/D3324

py3: iterate over a copy of dict while changing it Differential Revision: https://phab.mercurial-scm.org/D3323

py3: use urllib.parse.unquote_plus instead of urllib.unquote_plus The later is not present in Python 3. Differential Revision: https://phab.mercurial-scm.org/D3322

py3: add b'' prefixes to tests/test-status-inprocess.py # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3321

lock: don't use 'file' as a variable name Differential Revision: https://phab.mercurial-scm.org/D3320

py3: use b"%d" instead of str() to convert integers to bytes Differential Revision: https://phab.mercurial-scm.org/D3319

wireproto: expose repository formats via capabilities Servers need to expose their set of repository storage requirements in order to facilitate streaming clones (clients need to know if they are capable of reading the raw storage files that the server exposes). Differential Revision: https://phab.mercurial-scm.org/D3335

wireprotoframing: record when new stream is encountered Without this, we choke after receiving the 2nd frame in a stream. Not sure how we made it this far without finding this bug. Differential Revision: https://phab.mercurial-scm.org/D3334

wireprotoframing: use value passed into function Oops. Differential Revision: https://phab.mercurial-scm.org/D3333

httppeer: handle error response from client reactor With this in place, we're now seeing useful errors when running tests with the new wire protocol enabled! Differential Revision: https://phab.mercurial-scm.org/D3332

wireproto: add media type to version 2 capabilities response This is useful to advertise because servers reject unsupported media types. A client may wish to speak multiple media types and choose the one the server supports. I doubt we'll ever use multiple media types or negotiation in core. But during the course of developing this protocol, I may end up making extensions that backport and forward port protocol support as needed to support Mercurial deploys in the wild. e.g. I may deploy support for an older protocol on a server so old clients can continue using it. It's worth pursuing changing the SSH protocol's upgrade mechanism to support multiple media types as well... Differential Revision: https://phab.mercurial-scm.org/D3299

debugcommands: use command executor for invoking commands Now that all peers support the command executor interface, we can use it in place of peer._call() (which isn't part of the public API). Tests changed because we are now returning the decoded response instead of the raw bytes. The raw bytes over the wire are already logged. So we're not losing any test coverage. In fact, we're adding test coverage because we're testing decoding of those command responses as well. Differential Revision: https://phab.mercurial-scm.org/D3298

httppeer: implement command executor for version 2 peer Now that we have a new API for issuing commands which is compatible with wire protocol version 2, we can start using it with wire protocol version 2. This commit replaces our hacky implementation of _call() with something a bit more robust based on the new command executor interface. We now have proper support for issuing multiple commands per HTTP request. Each HTTP request maintains its own client reactor. The implementation is similar to the one in the legacy wire protocol. We use a ThreadPoolExecutor for spinning up a thread to read the HTTP response in the background. This allows responses to resolve in any order. While not implemented on the server yet, a client could use concurrent.futures.as_completed() with a collection of futures and handle responses as they arrive from the server. The return value from issued commands is still a simple list of raw or decoded CBOR data. This is still super hacky. We will want a rich data type for representing command responses. But at least this commit gets us one step closer to a proper peer implementation. Differential Revision: https://phab.mercurial-scm.org/D3297

repository: remove ipeercommands from ipeerbase AFAICT all callers in core have moved to the commandexecutor interface for invoking wire protocol commands. Or at least they aren't using the named methods on ipeercommands to invoke them. This means we can drop ipeercommands from the ipeerbase interface. As far as interface based programming goes, it is now illegal to call an ipeercommands method for issuing wire protocol commands. However, the methods are still there, so they will still work. At some point we will want to break that API... Differential Revision: https://phab.mercurial-scm.org/D3318

wireproto: properly call clonebundles command We should not be using _call() to make wire protocol calls because it isn't part of the peer API. But clonebundles wasn't part of the supported commands in the peer API! So this commit defines that command in the commands interface, implements it, and teaches the one caller in core to call it using the command executor interface. Differential Revision: https://phab.mercurial-scm.org/D3317

exchange: use command executor for getbundle The code consuming the bundle has been moved to inside the context manager, as that is supposed to be part of the API. (Although it doesn't matter for version 1 peers.) Differential Revision: https://phab.mercurial-scm.org/D3316

exchange: use command executor for pushkey Differential Revision: https://phab.mercurial-scm.org/D3315

wireproto: use command executor for unbundle This also required unifying the name of the argument because the new API always passes arguments by keyword. I decided to change implementations to "bundle" instead of the interface to "cg" because "bundle" is more appropriate in a modern world. Differential Revision: https://phab.mercurial-scm.org/D3314

debugcommands: perform handshake when obtaining httpv2 peer If we obtain an httpv2peer directly, the instance doesn't have an API descriptor and therefore doesn't know about the remote's commands, feature support, etc. This doesn't matter now. But when we implement the peer so it consults the API descriptor as part of sending commands, it will. So we change the logic for obtaining an http version 2 peer to go through makepeer() so the peer will perform the handshake and pass the API descriptor to the httpv2peer instance. Tests changed because we now perform a ?cmd=capabilities when obtaining version 2 peers. The Content-Length header is globbed because compression info will lack zstandard for pure builds. Differential Revision: https://phab.mercurial-scm.org/D3296

wireproto: rename HTTPV2 so it less like HTTP/2 Per review suggestion on D3230 from Augie. Differential Revision: https://phab.mercurial-scm.org/D3295

bundlerepo: use command executor for wire protocol commands Differential Revision: https://phab.mercurial-scm.org/D3294

bundlerepo: rename "other" to "peer" Differential Revision: https://phab.mercurial-scm.org/D3293

bookmarks: use command executor for wire protocol commands And change the name of a variable to reflect that is is a peer. Differential Revision: https://phab.mercurial-scm.org/D3292

hg: use command executor for wire protocol commands As part of this, I realized that some uses of lookup in a loop could be converted to use a batch request. But I didn't change behavior and left in a todo to track potentially changing this. Differential Revision: https://phab.mercurial-scm.org/D3291

logexchange: use command executor for wire protocol commands Differential Revision: https://phab.mercurial-scm.org/D3290

streamclone: use command executor for wire protocol commands Differential Revision: https://phab.mercurial-scm.org/D3289

discovery: use command executor interface We're trying to port all wire protocol code to use the new interface so we can implement wire protocol version 2 clients. Differential Revision: https://phab.mercurial-scm.org/D3288

discovery: don't redundantly call branchmap We were calling the remote command twice without mutation the remote in between. Derp. Differential Revision: https://phab.mercurial-scm.org/D3287

wireproto: convert legacy commands to command executor Calls to the legacy commands "changegroup" and "changegroupsubset" have been ported to the new command executor interface. Because we always pass arguments by name and not position, some inconsistent names throughout the code base have been unified. As part of this change, we no longer had any remaining callers of the legacy command methods {between, branches, changegroup, changegroupsubset}. So, these interfaces/methods have been dropped from peer interfaces. We still have an interface declaring these methods. But that interface is implemented on the concrete peer types and isn't part of the generic peer interface. (The implementations of the command executor continue to call these methods.) The ultimate goal is to remove the per-command methods from the generic peer interface: the only interface-conforming way to call a command will be with the new executor API. At some point, we may want to move the methods outside of the peer classes and change the executor implementations to not call methods directly on a peer instance. Differential Revision: https://phab.mercurial-scm.org/D3273

treediscovery: switch to command executor interface We now have a new interface for requesting that commands run. Switch to it. Differential Revision: https://phab.mercurial-scm.org/D3272

wireproto: remove iterbatch() from peer interface (API) Good riddance. Some tests have been ported to the new API. This probably should have been done in earlier commits. But duplicating the test coverage would have been difficult. It was easier this way. .. api:: The wire protocol peer's ``iterbatch()`` for bulk executing commands has been remove.d Use ``peer.commandexecutor()`` instead. Differential Revision: https://phab.mercurial-scm.org/D3271

largefiles: use command executor for batch operation This is the only other user of iterbatch() in core. Tests changed because the new command executor is smart enough to not send a "batch" command over the wire if only 1 command was requested. There is still coverage for the "batch" command in this test though. Differential Revision: https://phab.mercurial-scm.org/D3270

wireproto: implement batching on peer executor interface This is a bit more complicated than non-batch requests because we need to buffer sends until the last request arrives *and* we need to support resolving futures as data arrives from the remote. In a classical concurrent.futures executor model, the future "starts" as soon as it is submitted. However, we have nothing to start until the last command is submitted. If we did nothing, calling result() would deadlock, since the future hasn't "started." So in the case where we queue the command, we return a special future type whose result() will trigger sendcommands(). This eliminates the deadlock potential. It also serves as a check against callers who may be calling result() prematurely, as it will prevent any subsequent callcommands() from working. This behavior is slightly annoying and a bit restrictive. But it's the world that half duplex connections forces on us. In order to support streaming responses, we were previously using a generator. But with a futures-based API, we're using futures and not generators. So in order to get streaming, we need a background thread to read data from the server. The approach taken in this patch is to leverage the ThreadPoolExecutor from concurrent.futures for managing a background thread. We create an executor and future that resolves when all response data is processed (or an error occurs). When exiting the context manager, we wait on that background reading before returning. I was hoping we could manually spin up a threading.Thread and this would be simple. But I ran into a few deadlocks when implementing. After looking at the source code to concurrent.futures, I figured it would just be easier to use a ThreadPoolExecutor than implement all the code needed to manually manage a thread. To prove this works, a use of the batch API in discovery has been updated. Differential Revision: https://phab.mercurial-scm.org/D3269

wireproto: implement command executor interface for version 1 peers Now that we've defined our new interface for issuing commands, let's implement it. We add the interface to the base peer interface. This means all peer types must implement it. The only peer types that we have are the local peer in localrepo and a shared wire peer for version 1 of the wire protocol. The local peer implementation is pretty straightforward. We don't do anything fancy and just return a resolved future with the result of a method call. This is similar to what localiterbatcher does. The wire protocol version 1 implementation is a bit more complicated and is a more robust implementation. The wire executor queues commands by default. And because the new executor interface always allows multiple commands but not all version 1 commands are @batchable, it has to check that the requested commands are batchable if multiple commands are being requested. The wire executor currently only supports executing a single command. This is for simplicity reasons. Support for multiple commands will be added in a separate commit. To prove the new interface works, a call to the "known" command during discovery has been updated to use the new API. It's worth noting that both implementations require a method having the command name to exist on the peer. There is at least one caller in core that don't have a method calls peer._call() directly. We may need to shore up the requirements later... Differential Revision: https://phab.mercurial-scm.org/D3268

repository: define new interface for running commands Today, the peer interface exposes methods for each command that can be executed. In addition, there is an iterbatch() API that allows commands to be issued in batches and provides an iterator over the results. This is a glorified wrapper around the "batch" wire command. Wire protocol version 2 supports nicer things (such as batching any command and out-of-order replies). It will require a more flexible API for executing commands. This commit introduces a new peer interface for making command requests. In the new world, you can't simply call a method on the peer to execute a command: you need to obtain an object to be used for executing commands. That object can be used to issue a single command or it can batch multiple requests. In the case of full duplex peers, the command may even be sent out over the wire immediately. There are no per-command methods. Instead, there is a generic method to call a command. The implementation can then perform domain specific processing for specific commands. This includes passing data via a specially named argument. Arguments are also passed as a dictionary instead of using **kwargs. While **kwargs is nicer to use, we've historically gotten into trouble using it because there will inevitably be a conflict between the name of an argument to a wire protocol command and an argument we want to pass into a function. Instead of a command returning a value, it returns a future which will resolve to a value. This opens the door for out-of-order response handling and concurrent response handling in the version 2 protocol. Differential Revision: https://phab.mercurial-scm.org/D3267

pycompat: export a handle on concurrent.futures On Python 3, we use the built-in version in the standard library. Else we use our vendored backport. Differential Revision: https://phab.mercurial-scm.org/D3266

setup: add packages for concurrent.futures We conceivably don't need to distribute this package on Python 3 since we will use the version in the standard library. However, we want installs to be usable of multiple versions of Python. So it is best to always have it. Differential Revision: https://phab.mercurial-scm.org/D3265

futures: switch to absolute and relative imports This makes the package conform with our importing policy, silencing a number of warnings. It also makes the package usable when it isn't named "concurrent.futures." Differential Revision: https://phab.mercurial-scm.org/D3264

tests: silence pyflakes for thirdparty/concurrent It is complaining about unused imports. Differential Revision: https://phab.mercurial-scm.org/D3263

futures: get rid of extend_path This is used so mutliple directories can provide a package. We don't need it when vendoring. Differential Revision: https://phab.mercurial-scm.org/D3262

thirdparty: vendor futures 3.2.0 Python 3 has a concurrent.futures package in the standard library for representing futures. The "futures" package on PyPI is a backport of this package to work with Python 2. The wire protocol code today has its own future concept for handling of "batch" requests. The frame-based protocol will also want to use futures. I've heavily used the "futures" package on Python 2 in other projects and it is pretty nice. It even has a built-in thread and process pool for running functions in parallel. I've used this heavily for concurrent I/O and other GIL-less activities. The existing futures API in the wire protocol code is not as nice as concurrent.futures. Since concurrent.futures is in the Python standard library and will presumably be the long-term future for futures in our code base, let's vendor the backport so we can use proper futures today. # no-check-commit because of style violations Differential Revision: https://phab.mercurial-scm.org/D3261

py3: make sure decode() first argument is str Uses pycompat.sysstr() to make sure we uses bytes on Python 2 and unicodes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D3279

patch: make extract() a context manager (API) Previously, this function was creating a temporary file and relying on callers to unlink it. Yuck. We convert the function to a context manager and tie the lifetime of the temporary file to that of the context manager. This changed indentation not only from the context manager, but also from the elination of try blocks. It was just easier to split the heart of extract() into its own function. The single consumer of this function has been refactored to use it as a context manager. Code for cleaning up the file in tryimportone() has also been removed. .. api:: ``patch.extract()`` is now a context manager. Callers no longer have to worry about deleting the temporary file it creates, as the file is tied to the lifetime of the context manager. Differential Revision: https://phab.mercurial-scm.org/D3306

cmdutil: pass in parsed patch to tryimportone() (API) Previously, we parsed the patch in tryimportone(). This assumes the input is in a patch format that needs to be parsed. We want to support feeding in data from other formats. So let's let the caller handle the parsing. One wonky thing about patch parsing is that patch.extract() creates a temp file to hold the diffs and it is up to tryimportone() to unlink that temp file. I'll improve this in a subsequent commit. Differential Revision: https://phab.mercurial-scm.org/D3305

stringutil: support more types with pprint() bytearray wasn't working. Integers and floats were not being formatted. I /think/ %f is portable across both Python 2 and 3, as it should default to 6 decimal points on each. Differential Revision: https://phab.mercurial-scm.org/D3302

fix: port most of the way to python 3 Only most of the way because we now have to decide: if we want to keep the current .format() interface for the config in hgrc, we have to use unicodes to do formatting in Python 3, rather than bytes. I'm basically fine with that, so a follow-up patch will do so. Differential Revision: https://phab.mercurial-scm.org/D3300

lfs: teach the blob server to handle --prefix

hgweb: fallback to checking wsgireq.env for REPO_NAME for 3rd party hosting Starting with d7fd203e36cc, SCM Manager began to 404 any repository access. What's happening is that it is generating a python script that creates an hgweb application (not hgwebdir), and launches hgweb via wsgicgi. It must be setting REPO_NAME in the process environment before launching this script, which gets picked up and put into wsgireq.env when wsgicgi launches the hgweb application. >From there, other variables (notably 'apppath' and 'dispatchpath') are constructed differently. d7fd203e36cc^ (working): apppath: /hg/eng/devsetup dispatchpath: pathinfo: /eng/devsetup reponame: eng/devsetup d7fd203e36cc: apppath: /hg dispatchpath: eng/devsetup pathinfo: /eng/devsetup reponame: None REPO_NAME: eng/devsetup Rather than having an existing installation break when Mercurial is upgraded, just resume checking the environment. I have no idea how many other hosting solutions would break without restoring this.

peer: scatter module to the wind (API) peer.py hardly contained any code. The code it did contain was generic to the version 1 peer interface or specific to the local repository peer. So code has been moved to wireprotov1peer and localrepo, as appropriate. Differential Revision: https://phab.mercurial-scm.org/D3260

wireproto: move version 1 peer functionality to standalone module (API) wireproto.py contains code for both the client and the server. There *should* be a somewhat strong separation between the two. This commit extracts the client-side code from wireproto.py into a new module - wireprotov1peer. Differential Revision: https://phab.mercurial-scm.org/D3259

wireproto: move gboptsmap to wireprototypes and rename (API) This is also shared between client and server and will need to exist in a shared module when that code is split into different modules. Differential Revision: https://phab.mercurial-scm.org/D3258

wireproto: move value encoding functions to wireprototypes (API) These functions should live in the same place. I plan to separate client from server code in upcoming commits. wireprototypes is where we are putting shared code like this. Differential Revision: https://phab.mercurial-scm.org/D3257

httppeer: basic implementation of capabilities interface This is a bit crude. The capabilities mechanism for version 2 of the wire protocol is a bit different from version 1. And code in core is relying on strings passed to capable() matching strings advertised by the "capabilities" wire protocol command. I may refactor the internal checking mechanism to be a bit more abstract or based on interfaces. Time will tell... Differential Revision: https://phab.mercurial-scm.org/D3256

repository: split capabilities methods into separate interface So we can implement them without having to implement support for every command. Differential Revision: https://phab.mercurial-scm.org/D3255

httppeer: implement ipeerconnection This is low hanging fruit. We might as well start somewhere. Differential Revision: https://phab.mercurial-scm.org/D3254

py3: whitelist another six passing tests Differential Revision: https://phab.mercurial-scm.org/D3286

py3: whitelist another nine passing tests Differential Revision: https://phab.mercurial-scm.org/D3253

hgweb: use our forked wsgiheaders module instead of stdlib one Now we use bytes for headers, rather than native strings. Differential Revision: https://phab.mercurial-scm.org/D2854

wsgiheaders: import a bytes-ified fork of wsgiref.headers from cpython@46f5072 This will let us restore Python 3 compatibility for tests that do http things. Differential Revision: https://phab.mercurial-scm.org/D3245

export: enable formatter support (API) This change is basically the same as "hg cat". A formatter object is created by caller. .. api:: ``cmdutil.export()`` takes a formatter as an argument.

export: extract function to write patch to file object (API) This is common use case of cmdutil.export(), and we wouldn't want to handle formatter thingy everywhere. .. api:: The ``fp`` argument is removed from ``cmdutil.export()``. Use ``cmdutil.exportfile()`` instead.

export: port _exportsingle() to formatter Pass 'fm' instead of 'write', and use fm.plain(), fm.write(), etc. instead. The callers will be updated later.

export: serialize revisions to be exported per destination file Prepares for porting to the formatter API, where we can't simply append to existing files because JSON can't be streamed for example. The modemap hack is removed since cmdutil.export() was the only user. I also made the destination filename printed only once.

export: split cmdutil.export() to single-file and maybe-multiple-files cases Porting "hg export" to formatter is a bit hard because cmdutil.export() may append to files if the fntemplate is specified. This patch splits the hard part from the trivial case.

export: remove unused argument 'rev' from _exportsingle()

export: do not start pager if output will be written to file A copy of 3b569745af6c.

formatter: carry opts to file-based formatters by basefm This makes it slightly easier to port "hg export" to formatter.

formatter: remove unused private attributes from baseformatter No idea what they were for.

fix: add --all flag to fix non-public non-obsolete revisions Differential Revision: https://phab.mercurial-scm.org/D3213

fsmonitor: layer on another hack in bser.c for os.stat() compat (issue5811) It's unclear to me how these `bserobj_tuple` objects are used, other than as stat objects. This should fix fsmonitor in the wake of ffa3026d4196 and similar changes. I regret the hack here, but the code already has plenty of hg-specific hacks. :( It feels like we should be able to use int(result.st_mtime) globally, but that doesn't work. See issue4836 for a bug that was hard to track down relating to rounding behavior causing very subtle dirstate problems. Differential Revision: https://phab.mercurial-scm.org/D2939

tests: use `f --newer` instead of `stat -c` in test-fix.t Also increase sleep to two seconds so this test will likely pass on FAT32. Differential Revision: https://phab.mercurial-scm.org/D3252

tests: glob away fqdn wherever we print it These localhost instances are actually from a getfqdn call, which means on some of my test systems it comes out as localhost.localdomain or 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa. I'm tired of this, so let's glob it away. Differential Revision: https://phab.mercurial-scm.org/D3251

hgweb: use native strings when interfacing with stdlib headers We're still parsing the stdlib-provided headers here, so we need to tread carefully and use native strings. Yuck. Differential Revision: https://phab.mercurial-scm.org/D3250

wireprotoserver: headers are bytes for us internally, use bytes This re-fixes test-pull-http.t on Python 3. Probably many others as well. Differential Revision: https://phab.mercurial-scm.org/D3249

hgweb: put response headers back into str for Python 3 This fixes a lot of hanging tests on Python 3, because "Content-Length" was getting sent as "b'Content-Length'" (yes, really) and then clients would expect a close-is-end body instead of counting off a certain number of bytes. Differential Revision: https://phab.mercurial-scm.org/D3248

tests: load showstack in test-pull-http.t so network hangs are easier to find This also gives us some minimal "it loads" coverage on showstack, which I rather like. showstack doesn't work on Windows per mbarbison, so it's disabled there. I added this in service of debugging a hang introduced on Python 3 by revision a88d68dc3ee8. I'm still not sure what the problem there is, but this at least gives us a little bit of a chance to figure out what's going on. Differential Revision: https://phab.mercurial-scm.org/D3247

keepalive: rewrite readinto() to not use read() It turns out http.client on Python 3 sometimes uses readinto() in the implementation of read(). Unfortunately, Python 2 doesn't have a readinto() in httplib's client, so we have to support both codepaths. Subclassing is bad, folks. Differential Revision: https://phab.mercurial-scm.org/D3246

py3: use bytes() instead of str() on util.url() We internally deal with bytes and anything as string breaks things. Differential Revision: https://phab.mercurial-scm.org/D3285

py3: use stringutil.forcebytestr() to convert error messages to bytes Differential Revision: https://phab.mercurial-scm.org/D3284

py3: suppress the return value of write() in tests/test-subrepo-missing.t write() on Python 3 returns a value whereas does not return anything on Python 2. So we need to supress the value. Differential Revision: https://phab.mercurial-scm.org/D3283

py3: add b'' prefixes in tests/test-linerange.py This makes the test pass on Python 3. # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3282

py3: add b'' prefix to make the regex bytes # skip-blame because just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D3281

py3: use b'%d' to convert int to bytes instead of str() Differential Revision: https://phab.mercurial-scm.org/D3280

py3: use print as a function in test-convert-git.t Differential Revision: https://phab.mercurial-scm.org/D3278

py3: use '%d' for integers instead of '%s' Differential Revision: https://phab.mercurial-scm.org/D3277

py3: prevent transformer from adding b'' by adding r'' prefix These are cases where we need to use str, therefore we add r'' prefix. # skip-blame because just r'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3276

py3: use pycompat.{strkwargs|byteskwargs} in infinitepush Differential Revision: https://phab.mercurial-scm.org/D3275

py3: make sure we open file in bytes mode Differential Revision: https://phab.mercurial-scm.org/D3274

diffhelpers: be more tolerant for stripped empty lines of CRLF ending Exchange. It appears to trim lines containing only whitespace as well as converting LF to CRLF.

diffhelpers: make return value of testhunk() more Pythonic It's no longer C.

patch: error out if reached to EOF while reading hunk This was where out-of-bounds read occurred in old C extension.

diffhelpers: remove unused return value from fixnewline() and addlines()

diffhelpers: move out of pure package

diffhelpers: naming and whitespace cleanup

diffhelpers: remove C implementation in favor of pure Python version

patch: stop using cext.diffhelpers The C implementation has a couple of memory bugs, and lacks error handling which could lead to SEGV. I could fix them one by one (and I mostly finished that), but the performance gain provided by cext.diffhelper is quite low. Besides, diffhelpers.addlines() calls back Python, linereader.readline(), from the innermost loop. $ hg export -R mozilla-central 0:100 > patch $ ls -lh patch -rw-r--r-- 184M patch $ hg init repo && hg -R repo import patch --time --bypass (cext) time: real 34.970 secs (user 32.720+0.000 sys 2.230+0.000) (pure) time: real 35.950 secs (user 33.600+0.000 sys 2.330+0.000) So, let's simply use the pure Python implementation.

diffhelpers: port docstrings from cext to pure I'll remove the C implementation.

py3: get rid of character access from pure.diffhelpers 's' is a result of readline(), so 'c == "\n"' means 's == "\n"'.

lfs: handle paths that don't end with '/' when inferring the blob store While here, I also checked the lfs.url config directly instead of testing the scheme, as requested by Yuya.

lfs: infer the blob store URL from an explicit push dest or default-push Unlike pull, the blobs are uploaded within the exchange.push() window, so simply wrap it and swap in a properly configured remote store. The '_subtoppath' field shouldn't be available during this window, but give the passed path priority for clarity. At one point I hit an AttributeError in one of the convert tests when trying to save the original remote blobstore when the swap was run unconditionally. I wrapped it in a util.safehasattr(), but then today I wasn't able to reproduce it. But now the whole thing is tucked under the requirement guard because without the requirement, there are no blobs in the repo, even if the extension is loaded.