mail: fix debug print, which appears to have been broken for a long time I noticed this when trying to debug very mysterious dummysmtpd problems. It turns out you can set self.debuglevel to a number greater than 0 and have smtplib print helpful debug output, but this output on our side of things was broken. Differential Revision: https://phab.mercurial-scm.org/D4218

patchbomb: python 3 really wants those email addresses in unicode At this point, test-patchbomb-tls.t almost passes on Python 3, but I'm not really sure what the issue is: we just get a `abort: Connection unexpectedly closed` that doesn't make sense to me. Differential Revision: https://phab.mercurial-scm.org/D3957

mail: stop using the smtplib.SSLFakeFile and use socket.socket.makefile They're equivalent, and the latter is what Python 3.3 says to use in the release notes. Turns out it works on Python 2 as well. Differential Revision: https://phab.mercurial-scm.org/D3956

mail: modernize check for Python-with-TLS We used to be going indirectly through the socket module, but now we just check for the ssl module. Differential Revision: https://phab.mercurial-scm.org/D3955

mail: cope with Py3 unicode antics on email addresses Differential Revision: https://phab.mercurial-scm.org/D3954

mail: fix _encode to be more correct on Python 3 This code appears to be on the wrong side of the law in Python 2, at least some of the time. In Python 3, it's definitely wrong in places, but fortunately that's easy to fix. Differential Revision: https://phab.mercurial-scm.org/D3953

patchbomb: work around email module really wanting to write unicode data Differential Revision: https://phab.mercurial-scm.org/D3951

convert: don't drop missing or corrupt tag entries Cleaning up the tags file could be a useful feature in some cases, so maybe there should be a switch for this. However, the default hg -> hg convert tries to maintain identical hashes (thus convert.hg.saverev is off by default, but is on by default for other source types). It looks like _rewritesubstate() has a `continue` in it, and therefore a similar problem. I ran into this conversion divergence when a coworker "merged" two repositories by copy/pasting all of the files from the source repo and massaging the code, and forgetting to revert the .hg* files. That silently emptied the .hgtags file after the conversion. (This isn't the manifest node bug Yuya has been helping with- this occurred well after the bzr -> hg conversion and wasn't a merge commit, which made it extra puzzling. That bug is still an issue.)

changegroup: invert conditional and dedent I don't like else-less blocks that are indented for no reason. Differential Revision: https://phab.mercurial-scm.org/D4217

changegroup: capture base node instead of rev in delta request I want the API in storage to be defined in terms of nodes, not revision numbers. Differential Revision: https://phab.mercurial-scm.org/D4216

changegroup: introduce requests to define delta generation Currently, we iterate through each revision we will be producing a delta for then call into 1 of 2 functions for generating that delta. Deltas are emitted as we iterate. A problem with this model is that revision generation is tightly coupled to the changegroup code. And the storage layer needs to expose APIs like deltaparent() so changegroup delta generation can produce a delta with that knowledge. Another problem is that in this model, deltas can only be produced sequentially after the previous delta was produced and emitted. Some storage backends might be capable of producing deltas in parallel (e.g. if the changegroup deltas are cached somewhere). This commit aims to solve these problems by turning delta generation into a 2 phase implementation where the first phase determines info about all the deltas that need to be generated and the 2nd phase resolves those deltas. We introduce a "revisiondeltarequest" object that holds data about a to-be-generated delta. We perform a full pass over all revisions whose delta is to be generated and generate a "revisiondeltarequest" for each. Then we iterate over the "revisiondeltarequest" instances and derive a "revisiondelta" for each. This patch was quite large. In order to avoid even more churn, aspects of the implementation are less than ideal. e.g. we're recording revision numbers instead of nodes in a few places and we don't yet have a formal API for resolving an iterable of revisiondeltarequest instances. Things will be improved in subsequent commits. Unfortunately, this commit reduces performance substantially. For `hg perfchangegroupchangelog` on my hg repo: ! wall 1.512607 comb 1.510000 user 1.490000 sys 0.020000 (best of 7) ! wall 2.150863 comb 2.150000 user 2.150000 sys 0.000000 (best of 5) And for `hg bundle -t none-v2 -a` for the mozilla-unified repo: 178.32user 4.22system 3:02.59elapsed 190.97user 4.17system 3:15.19elapsed Some of this was attributed to changelog slowdown. `hg perfchangegroupchangelog` on mozilla-unified: ! wall 21.688715 comb 21.690000 user 21.570000 sys 0.120000 (best of 3) ! wall 25.683659 comb 25.680000 user 25.540000 sys 0.140000 (best of 3) Profiling seems to reveal that the changelog slowdown is due to reading changelog revisions multiple times. First in the linknode callback to resolve the set of files changed. Second in the delta generation. Before, we likely had hit the last revision cache in the revlog when doing delta generation since we performed that immediately after performing the linknode callback. I'm not exactly sure where the other ~8s are being spent. It might be from overhead of constructing a few million revisiondeltarequest objects. I'm OK with the regression for now because it is in service of a larger cause (storage abstraction). I'll try to profile later and claw back the performance. Differential Revision: https://phab.mercurial-scm.org/D4215

changegroup: refactor delta parent code We had recently abstracted the delta parent functions to facilitate extracting code from cgpacker. Now that we're in a better place, it is time to revisit the design. Changegroup version 1 requires that the previous node be used as the delta parent. Later versions allow any available node to be used as the base. In the case where an arbitrary parent can be used, the choice of a delta parent is best left in the hands of the storage backend. So it makes sense for the delta parent selection to be hidden away in the storage layer. This means deferring the choice of the delta parent selection function to as close to delta generation time as possible. This commit moves the delta selection logic to essentially just before delta generation. However, because changegroup version 1 limits what we can do, we have retained the ability to force a delta against the previous revision. As part of this, I realized that the ellipsis parent function was unused! That's because ellipsis mode always sends full revisions and not deltas. Differential Revision: https://phab.mercurial-scm.org/D4214

changegroup: differentiate between fulltext and diff based deltas Previously, revisiondelta encoded a delta and an optional prefix containing a delta header. The underlying code could populate the delta with either a real delta or a fulltext revision. Following the theme of wanting to defer serialization of revision data to the changegroup format as long as possible, it seems prudent for the revision delta instance to capture what type of data is being represented. This could possibly allow us to encode revision data differently in the future. But for the short term, it makes the behavior of a revisiondelta more explicit. Differential Revision: https://phab.mercurial-scm.org/D4213

changegroup: minor cleanups to deltagroup() Differential Revision: https://phab.mercurial-scm.org/D4212

changegroup: emit revisiondelta instances from deltagroup() By abstracting the concept of a delta group away from its serialization (the changegroup format), we make it easier to establish alternate serialization formats. We also make it possible to move aspects of delta group generation into the storage layer. This will allow storage to make decisions about e.g. delta parent choices without the changegroup code needing storage APIs to determine delta parents. We're still a bit of a way from there. Future commits will work towards that world. Differential Revision: https://phab.mercurial-scm.org/D4211

changegroup: move file chunk emission to generate() Same deal as manifests. We want to get to a point where we can emit data structures from deltagroup() and derive the raw changegroup data as late as possible. Differential Revision: https://phab.mercurial-scm.org/D4210

changegroup: move manifest chunk emission to generate() We want to get to a point where we can emit data structures from deltagroup() and derive the raw changegroup data as late as possible. Differential Revision: https://phab.mercurial-scm.org/D4209

changegroup: move size tracking and end of manifests to generate() Preparing for all the generate* functions to emit data structures instead of raw chunks. Differential Revision: https://phab.mercurial-scm.org/D4208

changegroup: emit delta group close chunk outside of deltagroup() I want to make deltagroup() emit data structures rather than serialized deltas. Upcoming commits will demonstrate why. Differential Revision: https://phab.mercurial-scm.org/D4207

changegroup: extract cgpacker.group() to standalone function It doesn't need to be part of the packer class. Differential Revision: https://phab.mercurial-scm.org/D4206

changegroup: pass all state into group() This will allow us to split it into a standalone function. Differential Revision: https://phab.mercurial-scm.org/D4205

changegroup: inline _prune() into call sites The functionality is pretty simple. As a bonus, _prune() had special code for the manifest case. We can now exclude this check from the file call site. Differential Revision: https://phab.mercurial-scm.org/D4199

changegroup: inline _packmanifests() into generatemanifests() It is relatively small. Every other generate*() calls group() directly. So the new code is consistent. Differential Revision: https://phab.mercurial-scm.org/D4198

changegroup: invert conditional and dedent I don't like seeing code that visually resembles the pyramid of doom. Differential Revision: https://phab.mercurial-scm.org/D4197

changegroup: make _revisiondeltanarrow() a standalone function It doesn't require any state on the packer. Everything impacting behavior is passed in as a function. So split it out, just like what was done for _revisiondeltanormal(). Differential Revision: https://phab.mercurial-scm.org/D4196

changegroup: pass state into _revisiondeltanarrow After this, the method no longer accesses self and can be split into a standalone function. Differential Revision: https://phab.mercurial-scm.org/D4195

changegroup: inline _close() Now that it doesn't clear self._clrevtolocalrev on every invocation and is a simple one-liner that calls another function, we can do away with this method and inline its content into all call sites. Differential Revision: https://phab.mercurial-scm.org/D4194

changegroup: pass clrevtolocalrev to each group clrevtolocalrev is a per-changegroup group mapping revisions to aid with shallow clone. Back when this functionality was implemented in an extension, this dict was added to the packer instance so monkeypatched functions could reference it there. Now that this code is part of core, we can pass the dict to each consumer properly so it doesn't have to live on the cgpacker instance. This commit does that. Differential Revision: https://phab.mercurial-scm.org/D4193

changegroup: combine _generatefiles() into generatefiles() These were split out in a06aab274aef as part of moving the narrow code into core. They don't need to be separate functions. Differential Revision: https://phab.mercurial-scm.org/D4192

changegroup: define linknodes callbacks in generatefiles() This is how it is done everywhere else. But the logic here is a bit more complex because shallow clone needs to reference the original linknode implementation. But at least now all function implementations are defined in the same place. Differential Revision: https://phab.mercurial-scm.org/D4191

changegroup: track changelog to manifest revision map explicitly Previously, self._nextclrevtolocalrev was only populated as part of the changelog lookup callback. But cgpacker._close() was looking at self._nextclrevtolocalrev on every invocation. Since self._nextclrevtolocalrev is for communicating the mapping of changelog revisions to manifest revisions, this commit refactors the code to make that explicit. The changelog state now stores this mapping. And after the changelog group is emitted, we update self._clrevtolocalrev with that dict. self._nextclrevtolocalrev is unused and has been deleted. Differential Revision: https://phab.mercurial-scm.org/D4190

changegroup: remove _clnodetorev cgpacker._clnodetorev is a glorified cache/index of changelog nodes to revision numbers. I'm not sure why it exists. Maybe performance? But its presence is making refactoring of this code more complicated than it needs to be. This commit removes the cache and replaces it with direct lookups against the changelog. If this cache was for performance reasons, we should be able to restore it easily enough... after the changegroup refactor is complete. Differential Revision: https://phab.mercurial-scm.org/D4189

changegroup: rename _fullnodes to _fullclnodes So it is obvious which nodes we are talking about. And sneak in a docs change to reflect that this variable is a set. Differential Revision: https://phab.mercurial-scm.org/D4188

changegroup: move part of _revisiondeltanarrow into group() Now all the logic for determining which delta generation code is called lives in a single function. Differential Revision: https://phab.mercurial-scm.org/D4187

changegroup: populate _clnodetorev as part of changelog linknode lookup The thing that matters is that self._clnodetorev is populated with changesets that are being sent. Back when this code was in an extension, it wasn't possible to monkeypatch the changelog lookup function. Now that the code is in core, we can move this code to where it logically belongs. Differential Revision: https://phab.mercurial-scm.org/D4186

tests: rename variables in revlog index parse test for clarity Now it's unambiguous which one is the expected value. c_res_{1,2} was also misleading a bit because in --pure mode we're testing the old slow Python version against the modern optimized Python version. Differential Revision: https://phab.mercurial-scm.org/D4180

tests: move assertion closer to want/got declarations in test-parseindex2.py I find this easier to understand. Differential Revision: https://phab.mercurial-scm.org/D4179

tests: move chunks of test-parseindex2.py to use unittest properly This doesn't touch the version-detection tests yet, because those are more involved. Differential Revision: https://phab.mercurial-scm.org/D4178

tests: fix up indent width in test-parseindex2.py Differential Revision: https://phab.mercurial-scm.org/D4177

tests: start moving test-parseindex2.py to a unittest Using 2-space indents in this revision to make the code motion easier to review. I'll fix it in the next commit. Differential Revision: https://phab.mercurial-scm.org/D4176

tests: port test-absorb-filefixupstate to Python 3 Mostly b prefixes, but also some isinstance() checks and a couple of maplist() instances. The test now passes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4175

absorb: port partway to Python 3 Use pycompat.maplist() in the one place that matters and use the default iterator of a dict instead of iterkeys(). Two new tests pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4174

localrepo: better error when a repo exists but we lack permissions Claiming "repository foo not found" when the repository does exist causes confusion regularly ("where is the typo?"). Differential Revision: https://phab.mercurial-scm.org/D4122

changegroup: extract _revisiondeltanormal() to standalone function It wasn't accessing anything important on the cgpacker that warranted it being a method instead of a function. Differential Revision: https://phab.mercurial-scm.org/D4142

changegroup: inline _revchunk() into group() _revchunk() was pretty minimal. I think having all the code for generating data composing the changegroup in one function makes things easier to understand. As part of the refactor, we now call the _revisiondelta* functions explicitly. This paves the road to refactor their argument signatures. Differential Revision: https://phab.mercurial-scm.org/D4141

changegroup: pass mfdicts properly With the narrow code part of core, the hacky pass-argument-via- attribute-on-self can be accomplished with a regular function argument. Differential Revision: https://phab.mercurial-scm.org/D4140

changegroup: pass sorted revisions into group() (API) Currently, group() receives a list of nodes and calls _sortgroup() to sort them and turn them into revs. Since the sorting behavior varies depending on the type of data being transferred, I think it makes sense to perform the sorting before group() is invoked. This commit extracts _sortgroup() to a pair of standalone functions. It then moves the calling of these functions to the 3 call sites of group(). group() now receives an iterable of revs instead of nodes. Differential Revision: https://phab.mercurial-scm.org/D4139

changegroup: pull _fileheader out of cgpacker It doesn't need any state from the packer. Differential Revision: https://phab.mercurial-scm.org/D4138

changegroup: factor changelogdone into an argument The variable was basically tracking whether the current operation is being performed against the changelog or something else. So let's just pass such a flag to everything that needs to access it. I'm still not a huge fan of building changelog awareness into low-level functions like revision delta generation. But passing an argument is strictly better than state on the packer instance. Differential Revision: https://phab.mercurial-scm.org/D4137

changegroup: record changelogdone after fully consuming its data Setting this as a side-effect of calling _close() is wonky. There's only one group for changelog data. So we can wait until after all data has been emitted before recording it. Differential Revision: https://phab.mercurial-scm.org/D4136

changegroup: key off changelogdone We use self._changelogdone for similar checks. Let's make things consistent. Differential Revision: https://phab.mercurial-scm.org/D4135

perf: call _generatechangelog() instead of group() Now that we have a separate function for generating just the changelog bits, the perf command should call it so it gets more accurate behavior. This changes the results of this command on my hg repo significantly: ! wall 1.390502 comb 1.390000 user 1.370000 sys 0.020000 (best of 8) ! wall 1.768750 comb 1.760000 user 1.760000 sys 0.000000 (best of 6) Profiling seems to reveal that ~20% of execution time is spent in progress bar accounting and printing! If we run with progress.disable=true: ! wall 1.639134 comb 1.650000 user 1.630000 sys 0.020000 (best of 7) A nice speedup. But profiling still shows a good chunk of time being spent in progress bar accounting code. The reason is that the progress bar is conditionally enabled via an argument to cgpacker.group(). The previous code in perf.py calling into group() did not enable the progress bar but _generatechangelog() always does. I think it is important for the perf* commands to capture real-world use cases. And this code always runs with an active progress bar. So the regression is acceptable. That being said, terminal printing performance can vary substantially. I don't think perf* commands should test terminal printing unless explicitly desired. So I've disabled progress bar printing in this command. Differential Revision: https://phab.mercurial-scm.org/D4134

changegroup: factor changelog chunk generation into own function We have separate functions for generating manifests and filelogs. Let's split changelog into its own function so things are consistent. As part of this, we refactor the code slightly. Before, the changelog linknode callback was updating state on variables inherited via a closure. Since the closure is now separate from generate(), we need to a way pass state between generate() and _generatechangelog(). The return value of _generatechangelog() is a 2-tuple where the first item is a dict containing accumulated state. We then alias some of its members into the scope of generate() to reduce code churn. I will be converting other functions to a similar pattern in future commits. Differential Revision: https://phab.mercurial-scm.org/D4133

changegroup: pass function to resolve delta parents into constructor Previously, _deltaparent() encapsulated the logic for all 3 delta parent modes of operation. The choice of delta parent is static for the lifetime of the packer and can be passed into the packer as a callable. So do that. Differential Revision: https://phab.mercurial-scm.org/D4132

changegroup: restore original behavior of _nextclrevtolocalrev 0548f696795b accidentally changed the behavior of cgpacker._close(). The old behavior moved _nextclrevtolocalrev to _clrevtolocalrev only when _nextclrevtolocalrev was present and then removed _nextclrevtolocalrev. The bad behavior performed this move then cleared _clrevtolocalrev because it was the same object as _nextclrevtolocalrev. This commit restores the previous behavior. Surprisingly, no tests changed as a result of this bad logic. I'm not sure why. Differential Revision: https://phab.mercurial-scm.org/D4155

py3: whitelist another test caught by the ratchet Differential Revision: https://phab.mercurial-scm.org/D4173

debugcommands: force import of fileset in debugfileset It looks like Python 3's lazy importer is better than Python 2's for this command, and as a result we had no symbols in the filesetlang symbol table, which resulted in some really mysterious test-fileset.t failures around withstatus optimizations. Inserting this explicit import and forcing its evaluation fixes the test failure. Differential Revision: https://phab.mercurial-scm.org/D4172

linelog: optimize replacelines The optimization to avoid calling `annotate` inside `replacelines` is significant for practical use patterns. Before this patch: hg perflinelogedits ! wall 6.778478 comb 6.710000 user 6.700000 sys 0.010000 (best of 3) After this patch: hg perflinelogedits ! wall 0.136573 comb 0.140000 user 0.130000 sys 0.010000 (best of 63) Differential Revision: https://phab.mercurial-scm.org/D4150

linelog: extract `len(self._program)` to a local function This is a micro optimization prepared for following changes where `len(self._program)` is used in a loop. Differential Revision: https://phab.mercurial-scm.org/D4149

perf: add a command to benchmark linelog edits The use pattern of creating a linelog is usually by calling "replacelines" multiple times. Add a command to benchmark it. Differential Revision: https://phab.mercurial-scm.org/D4148