peer: scatter module to the wind (API) peer.py hardly contained any code. The code it did contain was generic to the version 1 peer interface or specific to the local repository peer. So code has been moved to wireprotov1peer and localrepo, as appropriate. Differential Revision: https://phab.mercurial-scm.org/D3260

wireproto: move version 1 peer functionality to standalone module (API) wireproto.py contains code for both the client and the server. There *should* be a somewhat strong separation between the two. This commit extracts the client-side code from wireproto.py into a new module - wireprotov1peer. Differential Revision: https://phab.mercurial-scm.org/D3259

wireproto: move gboptsmap to wireprototypes and rename (API) This is also shared between client and server and will need to exist in a shared module when that code is split into different modules. Differential Revision: https://phab.mercurial-scm.org/D3258

wireproto: move value encoding functions to wireprototypes (API) These functions should live in the same place. I plan to separate client from server code in upcoming commits. wireprototypes is where we are putting shared code like this. Differential Revision: https://phab.mercurial-scm.org/D3257

httppeer: basic implementation of capabilities interface This is a bit crude. The capabilities mechanism for version 2 of the wire protocol is a bit different from version 1. And code in core is relying on strings passed to capable() matching strings advertised by the "capabilities" wire protocol command. I may refactor the internal checking mechanism to be a bit more abstract or based on interfaces. Time will tell... Differential Revision: https://phab.mercurial-scm.org/D3256

repository: split capabilities methods into separate interface So we can implement them without having to implement support for every command. Differential Revision: https://phab.mercurial-scm.org/D3255

httppeer: implement ipeerconnection This is low hanging fruit. We might as well start somewhere. Differential Revision: https://phab.mercurial-scm.org/D3254

py3: whitelist another six passing tests Differential Revision: https://phab.mercurial-scm.org/D3286

py3: whitelist another nine passing tests Differential Revision: https://phab.mercurial-scm.org/D3253

hgweb: use our forked wsgiheaders module instead of stdlib one Now we use bytes for headers, rather than native strings. Differential Revision: https://phab.mercurial-scm.org/D2854

wsgiheaders: import a bytes-ified fork of wsgiref.headers from cpython@46f5072 This will let us restore Python 3 compatibility for tests that do http things. Differential Revision: https://phab.mercurial-scm.org/D3245

export: enable formatter support (API) This change is basically the same as "hg cat". A formatter object is created by caller. .. api:: ``cmdutil.export()`` takes a formatter as an argument.

export: extract function to write patch to file object (API) This is common use case of cmdutil.export(), and we wouldn't want to handle formatter thingy everywhere. .. api:: The ``fp`` argument is removed from ``cmdutil.export()``. Use ``cmdutil.exportfile()`` instead.

export: port _exportsingle() to formatter Pass 'fm' instead of 'write', and use fm.plain(), fm.write(), etc. instead. The callers will be updated later.

export: serialize revisions to be exported per destination file Prepares for porting to the formatter API, where we can't simply append to existing files because JSON can't be streamed for example. The modemap hack is removed since cmdutil.export() was the only user. I also made the destination filename printed only once.

export: split cmdutil.export() to single-file and maybe-multiple-files cases Porting "hg export" to formatter is a bit hard because cmdutil.export() may append to files if the fntemplate is specified. This patch splits the hard part from the trivial case.

export: remove unused argument 'rev' from _exportsingle()

export: do not start pager if output will be written to file A copy of 3b569745af6c.

formatter: carry opts to file-based formatters by basefm This makes it slightly easier to port "hg export" to formatter.

formatter: remove unused private attributes from baseformatter No idea what they were for.

fix: add --all flag to fix non-public non-obsolete revisions Differential Revision: https://phab.mercurial-scm.org/D3213

fsmonitor: layer on another hack in bser.c for os.stat() compat (issue5811) It's unclear to me how these `bserobj_tuple` objects are used, other than as stat objects. This should fix fsmonitor in the wake of ffa3026d4196 and similar changes. I regret the hack here, but the code already has plenty of hg-specific hacks. :( It feels like we should be able to use int(result.st_mtime) globally, but that doesn't work. See issue4836 for a bug that was hard to track down relating to rounding behavior causing very subtle dirstate problems. Differential Revision: https://phab.mercurial-scm.org/D2939

tests: use `f --newer` instead of `stat -c` in test-fix.t Also increase sleep to two seconds so this test will likely pass on FAT32. Differential Revision: https://phab.mercurial-scm.org/D3252

tests: glob away fqdn wherever we print it These localhost instances are actually from a getfqdn call, which means on some of my test systems it comes out as localhost.localdomain or 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa. I'm tired of this, so let's glob it away. Differential Revision: https://phab.mercurial-scm.org/D3251

hgweb: use native strings when interfacing with stdlib headers We're still parsing the stdlib-provided headers here, so we need to tread carefully and use native strings. Yuck. Differential Revision: https://phab.mercurial-scm.org/D3250

wireprotoserver: headers are bytes for us internally, use bytes This re-fixes test-pull-http.t on Python 3. Probably many others as well. Differential Revision: https://phab.mercurial-scm.org/D3249

hgweb: put response headers back into str for Python 3 This fixes a lot of hanging tests on Python 3, because "Content-Length" was getting sent as "b'Content-Length'" (yes, really) and then clients would expect a close-is-end body instead of counting off a certain number of bytes. Differential Revision: https://phab.mercurial-scm.org/D3248

tests: load showstack in test-pull-http.t so network hangs are easier to find This also gives us some minimal "it loads" coverage on showstack, which I rather like. showstack doesn't work on Windows per mbarbison, so it's disabled there. I added this in service of debugging a hang introduced on Python 3 by revision a88d68dc3ee8. I'm still not sure what the problem there is, but this at least gives us a little bit of a chance to figure out what's going on. Differential Revision: https://phab.mercurial-scm.org/D3247

keepalive: rewrite readinto() to not use read() It turns out http.client on Python 3 sometimes uses readinto() in the implementation of read(). Unfortunately, Python 2 doesn't have a readinto() in httplib's client, so we have to support both codepaths. Subclassing is bad, folks. Differential Revision: https://phab.mercurial-scm.org/D3246

py3: use bytes() instead of str() on util.url() We internally deal with bytes and anything as string breaks things. Differential Revision: https://phab.mercurial-scm.org/D3285

py3: use stringutil.forcebytestr() to convert error messages to bytes Differential Revision: https://phab.mercurial-scm.org/D3284

py3: suppress the return value of write() in tests/test-subrepo-missing.t write() on Python 3 returns a value whereas does not return anything on Python 2. So we need to supress the value. Differential Revision: https://phab.mercurial-scm.org/D3283

py3: add b'' prefixes in tests/test-linerange.py This makes the test pass on Python 3. # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3282

py3: add b'' prefix to make the regex bytes # skip-blame because just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D3281

py3: use b'%d' to convert int to bytes instead of str() Differential Revision: https://phab.mercurial-scm.org/D3280

py3: use print as a function in test-convert-git.t Differential Revision: https://phab.mercurial-scm.org/D3278

py3: use '%d' for integers instead of '%s' Differential Revision: https://phab.mercurial-scm.org/D3277

py3: prevent transformer from adding b'' by adding r'' prefix These are cases where we need to use str, therefore we add r'' prefix. # skip-blame because just r'' prefixes Differential Revision: https://phab.mercurial-scm.org/D3276

py3: use pycompat.{strkwargs|byteskwargs} in infinitepush Differential Revision: https://phab.mercurial-scm.org/D3275

py3: make sure we open file in bytes mode Differential Revision: https://phab.mercurial-scm.org/D3274

diffhelpers: be more tolerant for stripped empty lines of CRLF ending Exchange. It appears to trim lines containing only whitespace as well as converting LF to CRLF.

diffhelpers: make return value of testhunk() more Pythonic It's no longer C.

patch: error out if reached to EOF while reading hunk This was where out-of-bounds read occurred in old C extension.

diffhelpers: remove unused return value from fixnewline() and addlines()

diffhelpers: move out of pure package

diffhelpers: naming and whitespace cleanup

diffhelpers: remove C implementation in favor of pure Python version

patch: stop using cext.diffhelpers The C implementation has a couple of memory bugs, and lacks error handling which could lead to SEGV. I could fix them one by one (and I mostly finished that), but the performance gain provided by cext.diffhelper is quite low. Besides, diffhelpers.addlines() calls back Python, linereader.readline(), from the innermost loop. $ hg export -R mozilla-central 0:100 > patch $ ls -lh patch -rw-r--r-- 184M patch $ hg init repo && hg -R repo import patch --time --bypass (cext) time: real 34.970 secs (user 32.720+0.000 sys 2.230+0.000) (pure) time: real 35.950 secs (user 33.600+0.000 sys 2.330+0.000) So, let's simply use the pure Python implementation.

diffhelpers: port docstrings from cext to pure I'll remove the C implementation.

py3: get rid of character access from pure.diffhelpers 's' is a result of readline(), so 'c == "\n"' means 's == "\n"'.

lfs: handle paths that don't end with '/' when inferring the blob store While here, I also checked the lfs.url config directly instead of testing the scheme, as requested by Yuya.

lfs: infer the blob store URL from an explicit push dest or default-push Unlike pull, the blobs are uploaded within the exchange.push() window, so simply wrap it and swap in a properly configured remote store. The '_subtoppath' field shouldn't be available during this window, but give the passed path priority for clarity. At one point I hit an AttributeError in one of the convert tests when trying to save the original remote blobstore when the swap was run unconditionally. I wrapped it in a util.safehasattr(), but then today I wasn't able to reproduce it. But now the whole thing is tucked under the requirement guard because without the requirement, there are no blobs in the repo, even if the extension is loaded.

lfs: infer the blob store URL from an explicit pull source I don't see any easier way to do this because the update part of `hg pull -u` happens outside exchange.pull(), and commands.postincoming() doesn't take a path. So (ab)use the mechanism used by subrepos to redirect where subrepos are pulled from when an explicit path is given. As a bonus, this should allow lfs blobs to be pulled into a subrepo when it is checked out. An explicit push path can be handled within exchange.push(). That can be done next, outside of this dirty hack.

lfs: special case the null:// usercache instead of treating it as a url The previous code worked on Windows, but not on Unix, and a pending patch's test failed. The url being used was something like "/tmp/.../client1/null://", courtesy of ui.configpath(). Looking at the doc comment, this seems like it's maybe not the right function to call (why should a relative cache path be expanded relative to the repo root or config file?), but largefiles has been using it since 8b8dd13295db (Oct 2011). It was introduced in 1b591f9b7fd2 (Jan 2011) without comment or callers. A grep over the whole history shows that only largefiles used it until lfs and infinitepush came along recently. It looks like if the `if not os.path.isabs(v) or "://" not in v` in configpath() is changed to an 'and', both Linux and Windows are happy. I'm guessing that "://" is to pick off URLs, so that seems reasonable. But I'm not sure why it isn't explicitly "file://", and I thought that "file://foo" is relative anyway. (At least, there are doctests for file:///tmp in util.url.) There is no mention of this setting in the help, but it is referenced on the wiki page for largefiles. (There's no mention that this is intended to be a URL, and the example uses an absolute path.) I don't want this blocking the rest of the lfs server discovery stuff. It was also wrong to allow a file:// URL here, but not in largefiles.

tests: add tests showing pulling from infinitepush works over wire The current tests in test-infinitepush-ci.t showed that `hg pull -r <rev>` does not work. Digging in code, I found that we have logic for pulling from bundlestore without having client side logic. This patch adds test demonstrating that pulling from bundlestore works when working over wire. Pulling from bundlestore when the peer is a localpeer still does not works. Differential Revision: https://phab.mercurial-scm.org/D3072

fix: use a portable python script instead of sed in test Differential Revision: https://phab.mercurial-scm.org/D2988

py3: use pycompat.bytestr() where repr in involved Differential Revision: https://phab.mercurial-scm.org/D3244

httppeer: support protocol upgrade With the new handshake defined and in place on the server, we can now implement it on the client. The HTTP handshake mechanism has been taught to add headers advertising its support for the new capabilities response. Response handling has been adjusted to allow CBOR responses through. And makepeer() has been taught to instantiate a mutually supported peer. The HTTPv2 peer class doesn't implement the full peer interface. So HTTPv2 is not yet usable as a peer. Like the server side, we support registering handlers for different API services. This allows extensions to easily implement API services and peers. A practical use case for this is to provide a previous implementation of the experimental version 2 wire protocol to a future version of Mercurial. We know there will be BC breaks after 4.6 ships. But someone could take the peer and server code from 4.6, drop it in an extension, and allow its use indefinitely. Differential Revision: https://phab.mercurial-scm.org/D3243

wireproto: define and implement HTTP handshake to upgrade protocol When clients connect to repositories over HTTP, they issue a request to the well-known URL "?cmd=capabilities" to fetch the repository capabilities. This is the handshake portion of the HTTP protocol. This commit defines a mechanism to use that HTTP request to return information about modern server features. If a client sends an X-HgUpgrade-* header containing a list of client-supported API names, the server responds with a response containing information about available services. This includes the normal capabilities string. So if the server doesn't support any newer services, the client can easily fall back. By advertising supported services from clients, server operators can see and log what client support exists in the wild. This will also help with debugging. The response contains the base path to API services. We know there are potential issues with the <repo>/api/ URL space conflicting with hgwebdir and subrepos. By making the API URL dynamic from the perspective of the client, the URL for APIs is not subject to backwards compatibility concerns - at least as long as a ?cmd=capabilities request is made. We've also defined the ``cbor`` client capability for the X-HgProto-* header. This MUST be sent in order to get the modern response from "?cmd=capabilities". During implementation, I initially always sent an application/mercurial-cbor response. However, the handshake mechanism will be more future compatible if the client is in charge of which formats to request. We already perform content negotiation from X-HgProto-*, so keying off this for the capabilities response feels appropriate. In addition, I initially used application/cbor. However, it is conceivable that a non-Mercurial server could serve application/cbor. To rule out this possibility, I've invented a new media type that is Mercurial specific and can't be confused for generic CBOR. Differential Revision: https://phab.mercurial-scm.org/D3242

httppeer: only advertise partial-pull if capabilities are known We don't need to be advertising client protocol parameters as part of the capabilities request during the handshake because nothing in version 1 of the wire protocol will use this data. i.e. the advertisement is wasteful. Differential Revision: https://phab.mercurial-scm.org/D3241