mail: be more cautious about bytes vs str for py3 compat It's suboptimal that we get a bytes on 2 and a unicode on 3, but it's easy to work with and I'm disinclined to change anything unless we start using some sort of type inferencer. Differential Revision: https://phab.mercurial-scm.org/D4233

patchbomb: use native strings when determining attachment disposition Differential Revision: https://phab.mercurial-scm.org/D4232

mail: always fall back to iso-8859-1 if us-ascii won't work (BC) It looks like this was a well-intentioned backwards compat hack for previewing the output of `hg email` in a stable way. Unfortunately I think this hack's time has come, because Python 3 does a much better job of ensuring it actually emits *valid* email messages. In particular, Python 2 would blindly trust us that the bytes we handed it were valid for the encoding we claimed, but Python 3 has some more sniff-tests that we end up failing. As a result, if we're going to print an email to the terminal, try us-ascii first, but if that fails go straight to iso-8859-1 which should be reasonably readable for ascii-compatible patch bodies. This *will* be a breaking change for ascii-incompatible textual patch content, but I don't think that's avoidable if we want to continue using the email library from the stdlib. .. bc:: Emails from the patchbomb extension will always be printed as though they are iso-8859-1 if they're not valid us-ascii. Previously, previewed emails were always claimed to be us-ascii and might contain invalid byte sequences. Differential Revision: https://phab.mercurial-scm.org/D4231

tests: put some Python 3 polish on inline Python invocations A couple of these became inline python sessions because they got more involved, and one of them is super-annoying and writes directly to sys.stdout.buffer because I just couldn't make it work any other way. Differential Revision: https://phab.mercurial-scm.org/D4230

patchbomb: don't unintentionally duplicate headers This dict-like of headers is only mostly dict-like: if you set a key that already exists, you it appends another value, rather than replacing the one that was already present. This fixes test-patchbomb-bookmark.t on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4229

patchbomb: use sysstrs when describing content-type Differential Revision: https://phab.mercurial-scm.org/D4228

patchbomb: use native strs for email header keys and values Differential Revision: https://phab.mercurial-scm.org/D4223

patchbomb: use email.encoders instead of email.Encoders No idea when this became a thing, but it exists for me in both 2.7 and 3.6. Differential Revision: https://phab.mercurial-scm.org/D4222

mail: properly handle email addresses typically being unicodes Visible progress in test-patchbomb.t on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4221

tests: fix Python3 issues in Python one-liners in test-patchbomb.t # skip-blame just byte prefixes Differential Revision: https://phab.mercurial-scm.org/D4220

dummysmtpd: accept additional kwargs from stdlib smtpd This was causing extremely mysterious failures because smtpd is implemented using asynchat, which in turn doesn't appear to do anything *remotely* helpful if an exception occurs. # no-check-commit because I'm editing a foo_bar function signature test-patchbomb-tls.t now passes in Python 3, but only because it doesn't check message bodies like test-patchbomb.t. test-patchbomb.t is *full* of doubled headers in the output (eg [0]) which seems like an odd failure mode. 0: @@ -141,12 +154,17 @@ MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit + MIME-Version: 1.0 + Content-Type: text/plain; charset="us-ascii" + Content-Transfer-Encoding: 7bit Subject: [PATCH] bookmark X-Mercurial-Node: 8dab2639fd35f1e337ad866c372a5c44f1064e3c X-Mercurial-Series-Index: 1 X-Mercurial-Series-Total: 1 Differential Revision: https://phab.mercurial-scm.org/D4219

mail: fix debug print, which appears to have been broken for a long time I noticed this when trying to debug very mysterious dummysmtpd problems. It turns out you can set self.debuglevel to a number greater than 0 and have smtplib print helpful debug output, but this output on our side of things was broken. Differential Revision: https://phab.mercurial-scm.org/D4218

patchbomb: python 3 really wants those email addresses in unicode At this point, test-patchbomb-tls.t almost passes on Python 3, but I'm not really sure what the issue is: we just get a `abort: Connection unexpectedly closed` that doesn't make sense to me. Differential Revision: https://phab.mercurial-scm.org/D3957

mail: stop using the smtplib.SSLFakeFile and use socket.socket.makefile They're equivalent, and the latter is what Python 3.3 says to use in the release notes. Turns out it works on Python 2 as well. Differential Revision: https://phab.mercurial-scm.org/D3956

mail: modernize check for Python-with-TLS We used to be going indirectly through the socket module, but now we just check for the ssl module. Differential Revision: https://phab.mercurial-scm.org/D3955

mail: cope with Py3 unicode antics on email addresses Differential Revision: https://phab.mercurial-scm.org/D3954

mail: fix _encode to be more correct on Python 3 This code appears to be on the wrong side of the law in Python 2, at least some of the time. In Python 3, it's definitely wrong in places, but fortunately that's easy to fix. Differential Revision: https://phab.mercurial-scm.org/D3953

patchbomb: work around email module really wanting to write unicode data Differential Revision: https://phab.mercurial-scm.org/D3951

convert: don't drop missing or corrupt tag entries Cleaning up the tags file could be a useful feature in some cases, so maybe there should be a switch for this. However, the default hg -> hg convert tries to maintain identical hashes (thus convert.hg.saverev is off by default, but is on by default for other source types). It looks like _rewritesubstate() has a `continue` in it, and therefore a similar problem. I ran into this conversion divergence when a coworker "merged" two repositories by copy/pasting all of the files from the source repo and massaging the code, and forgetting to revert the .hg* files. That silently emptied the .hgtags file after the conversion. (This isn't the manifest node bug Yuya has been helping with- this occurred well after the bzr -> hg conversion and wasn't a merge commit, which made it extra puzzling. That bug is still an issue.)

changegroup: invert conditional and dedent I don't like else-less blocks that are indented for no reason. Differential Revision: https://phab.mercurial-scm.org/D4217

changegroup: capture base node instead of rev in delta request I want the API in storage to be defined in terms of nodes, not revision numbers. Differential Revision: https://phab.mercurial-scm.org/D4216

changegroup: introduce requests to define delta generation Currently, we iterate through each revision we will be producing a delta for then call into 1 of 2 functions for generating that delta. Deltas are emitted as we iterate. A problem with this model is that revision generation is tightly coupled to the changegroup code. And the storage layer needs to expose APIs like deltaparent() so changegroup delta generation can produce a delta with that knowledge. Another problem is that in this model, deltas can only be produced sequentially after the previous delta was produced and emitted. Some storage backends might be capable of producing deltas in parallel (e.g. if the changegroup deltas are cached somewhere). This commit aims to solve these problems by turning delta generation into a 2 phase implementation where the first phase determines info about all the deltas that need to be generated and the 2nd phase resolves those deltas. We introduce a "revisiondeltarequest" object that holds data about a to-be-generated delta. We perform a full pass over all revisions whose delta is to be generated and generate a "revisiondeltarequest" for each. Then we iterate over the "revisiondeltarequest" instances and derive a "revisiondelta" for each. This patch was quite large. In order to avoid even more churn, aspects of the implementation are less than ideal. e.g. we're recording revision numbers instead of nodes in a few places and we don't yet have a formal API for resolving an iterable of revisiondeltarequest instances. Things will be improved in subsequent commits. Unfortunately, this commit reduces performance substantially. For `hg perfchangegroupchangelog` on my hg repo: ! wall 1.512607 comb 1.510000 user 1.490000 sys 0.020000 (best of 7) ! wall 2.150863 comb 2.150000 user 2.150000 sys 0.000000 (best of 5) And for `hg bundle -t none-v2 -a` for the mozilla-unified repo: 178.32user 4.22system 3:02.59elapsed 190.97user 4.17system 3:15.19elapsed Some of this was attributed to changelog slowdown. `hg perfchangegroupchangelog` on mozilla-unified: ! wall 21.688715 comb 21.690000 user 21.570000 sys 0.120000 (best of 3) ! wall 25.683659 comb 25.680000 user 25.540000 sys 0.140000 (best of 3) Profiling seems to reveal that the changelog slowdown is due to reading changelog revisions multiple times. First in the linknode callback to resolve the set of files changed. Second in the delta generation. Before, we likely had hit the last revision cache in the revlog when doing delta generation since we performed that immediately after performing the linknode callback. I'm not exactly sure where the other ~8s are being spent. It might be from overhead of constructing a few million revisiondeltarequest objects. I'm OK with the regression for now because it is in service of a larger cause (storage abstraction). I'll try to profile later and claw back the performance. Differential Revision: https://phab.mercurial-scm.org/D4215

changegroup: refactor delta parent code We had recently abstracted the delta parent functions to facilitate extracting code from cgpacker. Now that we're in a better place, it is time to revisit the design. Changegroup version 1 requires that the previous node be used as the delta parent. Later versions allow any available node to be used as the base. In the case where an arbitrary parent can be used, the choice of a delta parent is best left in the hands of the storage backend. So it makes sense for the delta parent selection to be hidden away in the storage layer. This means deferring the choice of the delta parent selection function to as close to delta generation time as possible. This commit moves the delta selection logic to essentially just before delta generation. However, because changegroup version 1 limits what we can do, we have retained the ability to force a delta against the previous revision. As part of this, I realized that the ellipsis parent function was unused! That's because ellipsis mode always sends full revisions and not deltas. Differential Revision: https://phab.mercurial-scm.org/D4214

changegroup: differentiate between fulltext and diff based deltas Previously, revisiondelta encoded a delta and an optional prefix containing a delta header. The underlying code could populate the delta with either a real delta or a fulltext revision. Following the theme of wanting to defer serialization of revision data to the changegroup format as long as possible, it seems prudent for the revision delta instance to capture what type of data is being represented. This could possibly allow us to encode revision data differently in the future. But for the short term, it makes the behavior of a revisiondelta more explicit. Differential Revision: https://phab.mercurial-scm.org/D4213

changegroup: minor cleanups to deltagroup() Differential Revision: https://phab.mercurial-scm.org/D4212

changegroup: emit revisiondelta instances from deltagroup() By abstracting the concept of a delta group away from its serialization (the changegroup format), we make it easier to establish alternate serialization formats. We also make it possible to move aspects of delta group generation into the storage layer. This will allow storage to make decisions about e.g. delta parent choices without the changegroup code needing storage APIs to determine delta parents. We're still a bit of a way from there. Future commits will work towards that world. Differential Revision: https://phab.mercurial-scm.org/D4211

changegroup: move file chunk emission to generate() Same deal as manifests. We want to get to a point where we can emit data structures from deltagroup() and derive the raw changegroup data as late as possible. Differential Revision: https://phab.mercurial-scm.org/D4210

changegroup: move manifest chunk emission to generate() We want to get to a point where we can emit data structures from deltagroup() and derive the raw changegroup data as late as possible. Differential Revision: https://phab.mercurial-scm.org/D4209

changegroup: move size tracking and end of manifests to generate() Preparing for all the generate* functions to emit data structures instead of raw chunks. Differential Revision: https://phab.mercurial-scm.org/D4208

changegroup: emit delta group close chunk outside of deltagroup() I want to make deltagroup() emit data structures rather than serialized deltas. Upcoming commits will demonstrate why. Differential Revision: https://phab.mercurial-scm.org/D4207

changegroup: extract cgpacker.group() to standalone function It doesn't need to be part of the packer class. Differential Revision: https://phab.mercurial-scm.org/D4206

changegroup: pass all state into group() This will allow us to split it into a standalone function. Differential Revision: https://phab.mercurial-scm.org/D4205