Fri, 10 Aug 2018 02:30:14 -0400 util: fix signature of setsockopt in socket observer
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 02:30:14 -0400] rev 39097
util: fix signature of setsockopt in socket observer The format string expected the result, callers all passed the result, but we didn't actually accept the result to this function. I swear I've looked at this code a dozen times trying to find the problem, and it was this simple all along. Differential Revision: https://phab.mercurial-scm.org/D4253
Fri, 10 Aug 2018 02:03:26 -0400 tests: port test-glog.t to Python 3 by using extracted printrevset
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 02:03:26 -0400] rev 39096
tests: port test-glog.t to Python 3 by using extracted printrevset Differential Revision: https://phab.mercurial-scm.org/D4252
Fri, 10 Aug 2018 02:02:40 -0400 tests: extract printrevset extension from test-glog-beautifygraph.t
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 02:02:40 -0400] rev 39095
tests: extract printrevset extension from test-glog-beautifygraph.t Differential Revision: https://phab.mercurial-scm.org/D4251
Fri, 10 Aug 2018 02:18:41 -0400 tests: add missing b'' prefix in test-check-code.t
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 02:18:41 -0400] rev 39094
tests: add missing b'' prefix in test-check-code.t It now passes on Python 3. # skip-blame b prefix Differential Revision: https://phab.mercurial-scm.org/D4250
Fri, 10 Aug 2018 02:17:50 -0400 contrib: have check-code look at files in latin1 instead of ascii
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 02:17:50 -0400] rev 39093
contrib: have check-code look at files in latin1 instead of ascii This way all files open. So far none of our patterns look like they'll care. Differential Revision: https://phab.mercurial-scm.org/D4249
Fri, 10 Aug 2018 00:37:24 -0400 tests: port inline extension in test for beautifygraph to Python 3
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 00:37:24 -0400] rev 39092
tests: port inline extension in test for beautifygraph to Python 3 test-glog-beautifygraph.t now passes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4247
Fri, 10 Aug 2018 00:08:06 -0400 beautifygraph: use slicing instead of subscripting on bytestr
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 00:08:06 -0400] rev 39091
beautifygraph: use slicing instead of subscripting on bytestr Differential Revision: https://phab.mercurial-scm.org/D4246
Fri, 10 Aug 2018 00:07:45 -0400 beautifygraph: use sysstr for checking encoding._wide
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 00:07:45 -0400] rev 39090
beautifygraph: use sysstr for checking encoding._wide # skip-blame just an r prefix Differential Revision: https://phab.mercurial-scm.org/D4245
Fri, 10 Aug 2018 01:42:01 -0400 stringutil: have buildrepr delegate to pprint for unknown types
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 01:42:01 -0400] rev 39089
stringutil: have buildrepr delegate to pprint for unknown types This stabilizes some revset reprs across Python 2 and 3. Differential Revision: https://phab.mercurial-scm.org/D4244
Fri, 10 Aug 2018 01:41:31 -0400 stringutil: teach pprint about sets
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 01:41:31 -0400] rev 39088
stringutil: teach pprint about sets This is the old (Python 2) way of printing sets. I actually prefer the Python 3 version of the repr, but this will result in less test churn in the short term. Differential Revision: https://phab.mercurial-scm.org/D4243
Fri, 10 Aug 2018 00:12:55 -0400 parser: replace bespoke _brepr with stringutil.pprint
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 00:12:55 -0400] rev 39087
parser: replace bespoke _brepr with stringutil.pprint Differential Revision: https://phab.mercurial-scm.org/D4242
Fri, 10 Aug 2018 02:06:46 -0400 mq: use stringutil.pprint instead of pycompat.byterepr
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 02:06:46 -0400] rev 39086
mq: use stringutil.pprint instead of pycompat.byterepr Resolves a few output changes on Python 3 in test-mq.t, and gets us two new passing tests. Differential Revision: https://phab.mercurial-scm.org/D4248
Thu, 09 Aug 2018 13:27:05 -0400 tests: remove expected output of test-parseindex2.py
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 13:27:05 -0400] rev 39085
tests: remove expected output of test-parseindex2.py Everything uses unittest idioms now, so this can go away. Differential Revision: https://phab.mercurial-scm.org/D4185
Thu, 09 Aug 2018 13:13:16 -0400 parsers: adjust pure-python version to mimic a3dacabd476b
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 13:13:16 -0400] rev 39084
parsers: adjust pure-python version to mimic a3dacabd476b This was caught by the tests, to my surprise. I'll also follow up with a test of the index[-1] behavior so we can be sure that remains consistent, as I think that currently has no coverage. Differential Revision: https://phab.mercurial-scm.org/D4183
Thu, 09 Aug 2018 13:34:33 -0400 tests: restore Python 3 compat in test-parseindex2.py
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 13:34:33 -0400] rev 39083
tests: restore Python 3 compat in test-parseindex2.py I think this was previously getting lucky in Python 3, and would have stacktraced if something failed. Now that failure messages are eagerly computed, this turned into a problem. Differential Revision: https://phab.mercurial-scm.org/D4182
Thu, 09 Aug 2018 13:26:28 -0400 tests: port remaining bits of test-parseindex2 to unittest asserts
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 13:26:28 -0400] rev 39082
tests: port remaining bits of test-parseindex2 to unittest asserts Differential Revision: https://phab.mercurial-scm.org/D4181
Thu, 09 Aug 2018 23:13:56 -0400 tests: allow more detailed error message from re.compile
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 23:13:56 -0400] rev 39081
tests: allow more detailed error message from re.compile test-grep.t now passes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4240
Thu, 09 Aug 2018 23:13:35 -0400 grep: include format string in columns list
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 23:13:35 -0400] rev 39080
grep: include format string in columns list This lets us correctly use %d as needed. Differential Revision: https://phab.mercurial-scm.org/D4239
Thu, 09 Aug 2018 23:13:01 -0400 grep: coerce username to bytestr, not str
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 23:13:01 -0400] rev 39079
grep: coerce username to bytestr, not str Differential Revision: https://phab.mercurial-scm.org/D4238
Thu, 09 Aug 2018 23:12:44 -0400 grep: difflib sequencematcher opcodes are native strs
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 23:12:44 -0400] rev 39078
grep: difflib sequencematcher opcodes are native strs Differential Revision: https://phab.mercurial-scm.org/D4237
Thu, 09 Aug 2018 21:59:32 -0400 tests: allow for a bonus newline in base64'd email payload
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 21:59:32 -0400] rev 39077
tests: allow for a bonus newline in base64'd email payload Python 3 adds this newline, which is harmless. test-patchbomb.t now passes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4234
Thu, 09 Aug 2018 21:58:43 -0400 mail: be more cautious about bytes vs str for py3 compat
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 21:58:43 -0400] rev 39076
mail: be more cautious about bytes vs str for py3 compat It's suboptimal that we get a bytes on 2 and a unicode on 3, but it's easy to work with and I'm disinclined to change anything unless we start using some sort of type inferencer. Differential Revision: https://phab.mercurial-scm.org/D4233
Thu, 09 Aug 2018 21:57:46 -0400 patchbomb: use native strings when determining attachment disposition
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 21:57:46 -0400] rev 39075
patchbomb: use native strings when determining attachment disposition Differential Revision: https://phab.mercurial-scm.org/D4232
Thu, 09 Aug 2018 21:04:15 -0400 mail: always fall back to iso-8859-1 if us-ascii won't work (BC)
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 21:04:15 -0400] rev 39074
mail: always fall back to iso-8859-1 if us-ascii won't work (BC) It looks like this was a well-intentioned backwards compat hack for previewing the output of `hg email` in a stable way. Unfortunately I think this hack's time has come, because Python 3 does a much better job of ensuring it actually emits *valid* email messages. In particular, Python 2 would blindly trust us that the bytes we handed it were valid for the encoding we claimed, but Python 3 has some more sniff-tests that we end up failing. As a result, if we're going to print an email to the terminal, try us-ascii first, but if that fails go straight to iso-8859-1 which should be reasonably readable for ascii-compatible patch bodies. This *will* be a breaking change for ascii-incompatible textual patch content, but I don't think that's avoidable if we want to continue using the email library from the stdlib. .. bc:: Emails from the patchbomb extension will always be printed as though they are iso-8859-1 if they're not valid us-ascii. Previously, previewed emails were always claimed to be us-ascii and might contain invalid byte sequences. Differential Revision: https://phab.mercurial-scm.org/D4231
Thu, 09 Aug 2018 20:57:27 -0400 tests: put some Python 3 polish on inline Python invocations
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 20:57:27 -0400] rev 39073
tests: put some Python 3 polish on inline Python invocations A couple of these became inline python sessions because they got more involved, and one of them is super-annoying and writes directly to sys.stdout.buffer because I just couldn't make it work any other way. Differential Revision: https://phab.mercurial-scm.org/D4230
Thu, 09 Aug 2018 19:39:33 -0400 patchbomb: don't unintentionally duplicate headers
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 19:39:33 -0400] rev 39072
patchbomb: don't unintentionally duplicate headers This dict-like of headers is only mostly dict-like: if you set a key that already exists, you it appends another value, rather than replacing the one that was already present. This fixes test-patchbomb-bookmark.t on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4229
Thu, 09 Aug 2018 19:37:57 -0400 patchbomb: use sysstrs when describing content-type
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 19:37:57 -0400] rev 39071
patchbomb: use sysstrs when describing content-type Differential Revision: https://phab.mercurial-scm.org/D4228
Thu, 09 Aug 2018 18:28:43 -0400 patchbomb: use native strs for email header keys and values
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 18:28:43 -0400] rev 39070
patchbomb: use native strs for email header keys and values Differential Revision: https://phab.mercurial-scm.org/D4223
Thu, 09 Aug 2018 18:28:19 -0400 patchbomb: use email.encoders instead of email.Encoders
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 18:28:19 -0400] rev 39069
patchbomb: use email.encoders instead of email.Encoders No idea when this became a thing, but it exists for me in both 2.7 and 3.6. Differential Revision: https://phab.mercurial-scm.org/D4222
Thu, 09 Aug 2018 18:27:36 -0400 mail: properly handle email addresses typically being unicodes
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 18:27:36 -0400] rev 39068
mail: properly handle email addresses typically being unicodes Visible progress in test-patchbomb.t on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4221
Thu, 09 Aug 2018 20:08:30 -0700 tests: fix Python3 issues in Python one-liners in test-patchbomb.t
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 20:08:30 -0700] rev 39067
tests: fix Python3 issues in Python one-liners in test-patchbomb.t # skip-blame just byte prefixes Differential Revision: https://phab.mercurial-scm.org/D4220
Thu, 09 Aug 2018 17:46:07 -0400 dummysmtpd: accept additional kwargs from stdlib smtpd
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 17:46:07 -0400] rev 39066
dummysmtpd: accept additional kwargs from stdlib smtpd This was causing extremely mysterious failures because smtpd is implemented using asynchat, which in turn doesn't appear to do anything *remotely* helpful if an exception occurs. # no-check-commit because I'm editing a foo_bar function signature test-patchbomb-tls.t now passes in Python 3, but only because it doesn't check message bodies like test-patchbomb.t. test-patchbomb.t is *full* of doubled headers in the output (eg [0]) which seems like an odd failure mode. 0: @@ -141,12 +154,17 @@ MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit + MIME-Version: 1.0 + Content-Type: text/plain; charset="us-ascii" + Content-Transfer-Encoding: 7bit Subject: [PATCH] bookmark X-Mercurial-Node: 8dab2639fd35f1e337ad866c372a5c44f1064e3c X-Mercurial-Series-Index: 1 X-Mercurial-Series-Total: 1 Differential Revision: https://phab.mercurial-scm.org/D4219
Thu, 09 Aug 2018 17:44:36 -0400 mail: fix debug print, which appears to have been broken for a long time
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 17:44:36 -0400] rev 39065
mail: fix debug print, which appears to have been broken for a long time I noticed this when trying to debug very mysterious dummysmtpd problems. It turns out you can set self.debuglevel to a number greater than 0 and have smtplib print helpful debug output, but this output on our side of things was broken. Differential Revision: https://phab.mercurial-scm.org/D4218
Mon, 16 Jul 2018 18:17:44 -0400 patchbomb: python 3 really wants those email addresses in unicode
Augie Fackler <augie@google.com> [Mon, 16 Jul 2018 18:17:44 -0400] rev 39064
patchbomb: python 3 really wants those email addresses in unicode At this point, test-patchbomb-tls.t almost passes on Python 3, but I'm not really sure what the issue is: we just get a `abort: Connection unexpectedly closed` that doesn't make sense to me. Differential Revision: https://phab.mercurial-scm.org/D3957
Mon, 16 Jul 2018 18:17:19 -0400 mail: stop using the smtplib.SSLFakeFile and use socket.socket.makefile
Augie Fackler <augie@google.com> [Mon, 16 Jul 2018 18:17:19 -0400] rev 39063
mail: stop using the smtplib.SSLFakeFile and use socket.socket.makefile They're equivalent, and the latter is what Python 3.3 says to use in the release notes. Turns out it works on Python 2 as well. Differential Revision: https://phab.mercurial-scm.org/D3956
Mon, 16 Jul 2018 18:16:26 -0400 mail: modernize check for Python-with-TLS
Augie Fackler <augie@google.com> [Mon, 16 Jul 2018 18:16:26 -0400] rev 39062
mail: modernize check for Python-with-TLS We used to be going indirectly through the socket module, but now we just check for the ssl module. Differential Revision: https://phab.mercurial-scm.org/D3955
Mon, 16 Jul 2018 17:49:17 -0400 mail: cope with Py3 unicode antics on email addresses
Augie Fackler <augie@google.com> [Mon, 16 Jul 2018 17:49:17 -0400] rev 39061
mail: cope with Py3 unicode antics on email addresses Differential Revision: https://phab.mercurial-scm.org/D3954
Mon, 16 Jul 2018 17:48:03 -0400 mail: fix _encode to be more correct on Python 3
Augie Fackler <augie@google.com> [Mon, 16 Jul 2018 17:48:03 -0400] rev 39060
mail: fix _encode to be more correct on Python 3 This code appears to be on the wrong side of the law in Python 2, at least some of the time. In Python 3, it's definitely wrong in places, but fortunately that's easy to fix. Differential Revision: https://phab.mercurial-scm.org/D3953
Mon, 16 Jul 2018 14:14:27 -0400 patchbomb: work around email module really wanting to write unicode data
Augie Fackler <augie@google.com> [Mon, 16 Jul 2018 14:14:27 -0400] rev 39059
patchbomb: work around email module really wanting to write unicode data Differential Revision: https://phab.mercurial-scm.org/D3951
Thu, 09 Aug 2018 12:25:06 -0700 changegroup: invert conditional and dedent
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 09 Aug 2018 12:25:06 -0700] rev 39058
changegroup: invert conditional and dedent I don't like else-less blocks that are indented for no reason. Differential Revision: https://phab.mercurial-scm.org/D4217
Thu, 09 Aug 2018 09:40:29 -0700 changegroup: capture base node instead of rev in delta request
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 09 Aug 2018 09:40:29 -0700] rev 39057
changegroup: capture base node instead of rev in delta request I want the API in storage to be defined in terms of nodes, not revision numbers. Differential Revision: https://phab.mercurial-scm.org/D4216
Thu, 09 Aug 2018 09:28:26 -0700 changegroup: introduce requests to define delta generation
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 09 Aug 2018 09:28:26 -0700] rev 39056
changegroup: introduce requests to define delta generation Currently, we iterate through each revision we will be producing a delta for then call into 1 of 2 functions for generating that delta. Deltas are emitted as we iterate. A problem with this model is that revision generation is tightly coupled to the changegroup code. And the storage layer needs to expose APIs like deltaparent() so changegroup delta generation can produce a delta with that knowledge. Another problem is that in this model, deltas can only be produced sequentially after the previous delta was produced and emitted. Some storage backends might be capable of producing deltas in parallel (e.g. if the changegroup deltas are cached somewhere). This commit aims to solve these problems by turning delta generation into a 2 phase implementation where the first phase determines info about all the deltas that need to be generated and the 2nd phase resolves those deltas. We introduce a "revisiondeltarequest" object that holds data about a to-be-generated delta. We perform a full pass over all revisions whose delta is to be generated and generate a "revisiondeltarequest" for each. Then we iterate over the "revisiondeltarequest" instances and derive a "revisiondelta" for each. This patch was quite large. In order to avoid even more churn, aspects of the implementation are less than ideal. e.g. we're recording revision numbers instead of nodes in a few places and we don't yet have a formal API for resolving an iterable of revisiondeltarequest instances. Things will be improved in subsequent commits. Unfortunately, this commit reduces performance substantially. For `hg perfchangegroupchangelog` on my hg repo: ! wall 1.512607 comb 1.510000 user 1.490000 sys 0.020000 (best of 7) ! wall 2.150863 comb 2.150000 user 2.150000 sys 0.000000 (best of 5) And for `hg bundle -t none-v2 -a` for the mozilla-unified repo: 178.32user 4.22system 3:02.59elapsed 190.97user 4.17system 3:15.19elapsed Some of this was attributed to changelog slowdown. `hg perfchangegroupchangelog` on mozilla-unified: ! wall 21.688715 comb 21.690000 user 21.570000 sys 0.120000 (best of 3) ! wall 25.683659 comb 25.680000 user 25.540000 sys 0.140000 (best of 3) Profiling seems to reveal that the changelog slowdown is due to reading changelog revisions multiple times. First in the linknode callback to resolve the set of files changed. Second in the delta generation. Before, we likely had hit the last revision cache in the revlog when doing delta generation since we performed that immediately after performing the linknode callback. I'm not exactly sure where the other ~8s are being spent. It might be from overhead of constructing a few million revisiondeltarequest objects. I'm OK with the regression for now because it is in service of a larger cause (storage abstraction). I'll try to profile later and claw back the performance. Differential Revision: https://phab.mercurial-scm.org/D4215
Wed, 08 Aug 2018 20:17:48 -0700 changegroup: refactor delta parent code
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 20:17:48 -0700] rev 39055
changegroup: refactor delta parent code We had recently abstracted the delta parent functions to facilitate extracting code from cgpacker. Now that we're in a better place, it is time to revisit the design. Changegroup version 1 requires that the previous node be used as the delta parent. Later versions allow any available node to be used as the base. In the case where an arbitrary parent can be used, the choice of a delta parent is best left in the hands of the storage backend. So it makes sense for the delta parent selection to be hidden away in the storage layer. This means deferring the choice of the delta parent selection function to as close to delta generation time as possible. This commit moves the delta selection logic to essentially just before delta generation. However, because changegroup version 1 limits what we can do, we have retained the ability to force a delta against the previous revision. As part of this, I realized that the ellipsis parent function was unused! That's because ellipsis mode always sends full revisions and not deltas. Differential Revision: https://phab.mercurial-scm.org/D4214
Wed, 08 Aug 2018 16:01:26 -0700 changegroup: differentiate between fulltext and diff based deltas
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 16:01:26 -0700] rev 39054
changegroup: differentiate between fulltext and diff based deltas Previously, revisiondelta encoded a delta and an optional prefix containing a delta header. The underlying code could populate the delta with either a real delta or a fulltext revision. Following the theme of wanting to defer serialization of revision data to the changegroup format as long as possible, it seems prudent for the revision delta instance to capture what type of data is being represented. This could possibly allow us to encode revision data differently in the future. But for the short term, it makes the behavior of a revisiondelta more explicit. Differential Revision: https://phab.mercurial-scm.org/D4213
Wed, 08 Aug 2018 15:28:22 -0700 changegroup: minor cleanups to deltagroup()
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 15:28:22 -0700] rev 39053
changegroup: minor cleanups to deltagroup() Differential Revision: https://phab.mercurial-scm.org/D4212
Wed, 08 Aug 2018 14:44:48 -0700 changegroup: emit revisiondelta instances from deltagroup()
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 14:44:48 -0700] rev 39052
changegroup: emit revisiondelta instances from deltagroup() By abstracting the concept of a delta group away from its serialization (the changegroup format), we make it easier to establish alternate serialization formats. We also make it possible to move aspects of delta group generation into the storage layer. This will allow storage to make decisions about e.g. delta parent choices without the changegroup code needing storage APIs to determine delta parents. We're still a bit of a way from there. Future commits will work towards that world. Differential Revision: https://phab.mercurial-scm.org/D4211
Wed, 08 Aug 2018 14:33:33 -0700 changegroup: move file chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 14:33:33 -0700] rev 39051
changegroup: move file chunk emission to generate() Same deal as manifests. We want to get to a point where we can emit data structures from deltagroup() and derive the raw changegroup data as late as possible. Differential Revision: https://phab.mercurial-scm.org/D4210
Wed, 08 Aug 2018 15:14:59 -0700 changegroup: move manifest chunk emission to generate()
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 15:14:59 -0700] rev 39050
changegroup: move manifest chunk emission to generate() We want to get to a point where we can emit data structures from deltagroup() and derive the raw changegroup data as late as possible. Differential Revision: https://phab.mercurial-scm.org/D4209
Wed, 08 Aug 2018 15:09:12 -0700 changegroup: move size tracking and end of manifests to generate()
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 15:09:12 -0700] rev 39049
changegroup: move size tracking and end of manifests to generate() Preparing for all the generate* functions to emit data structures instead of raw chunks. Differential Revision: https://phab.mercurial-scm.org/D4208
Wed, 08 Aug 2018 14:15:33 -0700 changegroup: emit delta group close chunk outside of deltagroup()
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 14:15:33 -0700] rev 39048
changegroup: emit delta group close chunk outside of deltagroup() I want to make deltagroup() emit data structures rather than serialized deltas. Upcoming commits will demonstrate why. Differential Revision: https://phab.mercurial-scm.org/D4207
Wed, 08 Aug 2018 14:19:02 -0700 changegroup: extract cgpacker.group() to standalone function
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 14:19:02 -0700] rev 39047
changegroup: extract cgpacker.group() to standalone function It doesn't need to be part of the packer class. Differential Revision: https://phab.mercurial-scm.org/D4206
Wed, 08 Aug 2018 14:02:31 -0700 changegroup: pass all state into group()
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 14:02:31 -0700] rev 39046
changegroup: pass all state into group() This will allow us to split it into a standalone function. Differential Revision: https://phab.mercurial-scm.org/D4205
Wed, 08 Aug 2018 13:50:54 -0700 changegroup: inline _prune() into call sites
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 13:50:54 -0700] rev 39045
changegroup: inline _prune() into call sites The functionality is pretty simple. As a bonus, _prune() had special code for the manifest case. We can now exclude this check from the file call site. Differential Revision: https://phab.mercurial-scm.org/D4199
Tue, 07 Aug 2018 15:31:03 -0700 changegroup: inline _packmanifests() into generatemanifests()
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 15:31:03 -0700] rev 39044
changegroup: inline _packmanifests() into generatemanifests() It is relatively small. Every other generate*() calls group() directly. So the new code is consistent. Differential Revision: https://phab.mercurial-scm.org/D4198
Tue, 07 Aug 2018 15:13:25 -0700 changegroup: invert conditional and dedent
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 15:13:25 -0700] rev 39043
changegroup: invert conditional and dedent I don't like seeing code that visually resembles the pyramid of doom. Differential Revision: https://phab.mercurial-scm.org/D4197
Tue, 07 Aug 2018 15:10:38 -0700 changegroup: make _revisiondeltanarrow() a standalone function
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 15:10:38 -0700] rev 39042
changegroup: make _revisiondeltanarrow() a standalone function It doesn't require any state on the packer. Everything impacting behavior is passed in as a function. So split it out, just like what was done for _revisiondeltanormal(). Differential Revision: https://phab.mercurial-scm.org/D4196
Tue, 07 Aug 2018 15:08:29 -0700 changegroup: pass state into _revisiondeltanarrow
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 15:08:29 -0700] rev 39041
changegroup: pass state into _revisiondeltanarrow After this, the method no longer accesses self and can be split into a standalone function. Differential Revision: https://phab.mercurial-scm.org/D4195
Tue, 07 Aug 2018 14:53:42 -0700 changegroup: inline _close()
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 14:53:42 -0700] rev 39040
changegroup: inline _close() Now that it doesn't clear self._clrevtolocalrev on every invocation and is a simple one-liner that calls another function, we can do away with this method and inline its content into all call sites. Differential Revision: https://phab.mercurial-scm.org/D4194
Tue, 07 Aug 2018 14:52:00 -0700 changegroup: pass clrevtolocalrev to each group
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 14:52:00 -0700] rev 39039
changegroup: pass clrevtolocalrev to each group clrevtolocalrev is a per-changegroup group mapping revisions to aid with shallow clone. Back when this functionality was implemented in an extension, this dict was added to the packer instance so monkeypatched functions could reference it there. Now that this code is part of core, we can pass the dict to each consumer properly so it doesn't have to live on the cgpacker instance. This commit does that. Differential Revision: https://phab.mercurial-scm.org/D4193
Tue, 07 Aug 2018 12:44:56 -0700 changegroup: combine _generatefiles() into generatefiles()
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 12:44:56 -0700] rev 39038
changegroup: combine _generatefiles() into generatefiles() These were split out in a06aab274aef as part of moving the narrow code into core. They don't need to be separate functions. Differential Revision: https://phab.mercurial-scm.org/D4192
Tue, 07 Aug 2018 15:45:56 -0700 changegroup: define linknodes callbacks in generatefiles()
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 15:45:56 -0700] rev 39037
changegroup: define linknodes callbacks in generatefiles() This is how it is done everywhere else. But the logic here is a bit more complex because shallow clone needs to reference the original linknode implementation. But at least now all function implementations are defined in the same place. Differential Revision: https://phab.mercurial-scm.org/D4191
Tue, 07 Aug 2018 10:55:32 -0700 changegroup: track changelog to manifest revision map explicitly
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 10:55:32 -0700] rev 39036
changegroup: track changelog to manifest revision map explicitly Previously, self._nextclrevtolocalrev was only populated as part of the changelog lookup callback. But cgpacker._close() was looking at self._nextclrevtolocalrev on every invocation. Since self._nextclrevtolocalrev is for communicating the mapping of changelog revisions to manifest revisions, this commit refactors the code to make that explicit. The changelog state now stores this mapping. And after the changelog group is emitted, we update self._clrevtolocalrev with that dict. self._nextclrevtolocalrev is unused and has been deleted. Differential Revision: https://phab.mercurial-scm.org/D4190
Tue, 07 Aug 2018 10:49:41 -0700 changegroup: remove _clnodetorev
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 10:49:41 -0700] rev 39035
changegroup: remove _clnodetorev cgpacker._clnodetorev is a glorified cache/index of changelog nodes to revision numbers. I'm not sure why it exists. Maybe performance? But its presence is making refactoring of this code more complicated than it needs to be. This commit removes the cache and replaces it with direct lookups against the changelog. If this cache was for performance reasons, we should be able to restore it easily enough... after the changegroup refactor is complete. Differential Revision: https://phab.mercurial-scm.org/D4189
Mon, 06 Aug 2018 15:44:33 -0700 changegroup: rename _fullnodes to _fullclnodes
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 15:44:33 -0700] rev 39034
changegroup: rename _fullnodes to _fullclnodes So it is obvious which nodes we are talking about. And sneak in a docs change to reflect that this variable is a set. Differential Revision: https://phab.mercurial-scm.org/D4188
Mon, 06 Aug 2018 15:04:20 -0700 changegroup: move part of _revisiondeltanarrow into group()
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 15:04:20 -0700] rev 39033
changegroup: move part of _revisiondeltanarrow into group() Now all the logic for determining which delta generation code is called lives in a single function. Differential Revision: https://phab.mercurial-scm.org/D4187
Mon, 06 Aug 2018 14:56:37 -0700 changegroup: populate _clnodetorev as part of changelog linknode lookup
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 14:56:37 -0700] rev 39032
changegroup: populate _clnodetorev as part of changelog linknode lookup The thing that matters is that self._clnodetorev is populated with changesets that are being sent. Back when this code was in an extension, it wasn't possible to monkeypatch the changelog lookup function. Now that the code is in core, we can move this code to where it logically belongs. Differential Revision: https://phab.mercurial-scm.org/D4186
Thu, 09 Aug 2018 13:08:29 -0400 tests: rename variables in revlog index parse test for clarity
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 13:08:29 -0400] rev 39031
tests: rename variables in revlog index parse test for clarity Now it's unambiguous which one is the expected value. c_res_{1,2} was also misleading a bit because in --pure mode we're testing the old slow Python version against the modern optimized Python version. Differential Revision: https://phab.mercurial-scm.org/D4180
Thu, 09 Aug 2018 13:06:50 -0400 tests: move assertion closer to want/got declarations in test-parseindex2.py
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 13:06:50 -0400] rev 39030
tests: move assertion closer to want/got declarations in test-parseindex2.py I find this easier to understand. Differential Revision: https://phab.mercurial-scm.org/D4179
Thu, 09 Aug 2018 13:05:40 -0400 tests: move chunks of test-parseindex2.py to use unittest properly
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 13:05:40 -0400] rev 39029
tests: move chunks of test-parseindex2.py to use unittest properly This doesn't touch the version-detection tests yet, because those are more involved. Differential Revision: https://phab.mercurial-scm.org/D4178
Thu, 09 Aug 2018 12:59:23 -0400 tests: fix up indent width in test-parseindex2.py
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 12:59:23 -0400] rev 39028
tests: fix up indent width in test-parseindex2.py Differential Revision: https://phab.mercurial-scm.org/D4177
Thu, 09 Aug 2018 12:58:25 -0400 tests: start moving test-parseindex2.py to a unittest
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 12:58:25 -0400] rev 39027
tests: start moving test-parseindex2.py to a unittest Using 2-space indents in this revision to make the code motion easier to review. I'll fix it in the next commit. Differential Revision: https://phab.mercurial-scm.org/D4176
Thu, 09 Aug 2018 12:10:34 -0400 tests: port test-absorb-filefixupstate to Python 3
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 12:10:34 -0400] rev 39026
tests: port test-absorb-filefixupstate to Python 3 Mostly b prefixes, but also some isinstance() checks and a couple of maplist() instances. The test now passes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4175
Thu, 09 Aug 2018 12:06:31 -0400 absorb: port partway to Python 3
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 12:06:31 -0400] rev 39025
absorb: port partway to Python 3 Use pycompat.maplist() in the one place that matters and use the default iterator of a dict instead of iterkeys(). Two new tests pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4174
Sat, 04 Aug 2018 21:31:46 -0400 localrepo: better error when a repo exists but we lack permissions
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Sat, 04 Aug 2018 21:31:46 -0400] rev 39024
localrepo: better error when a repo exists but we lack permissions Claiming "repository foo not found" when the repository does exist causes confusion regularly ("where is the typo?"). Differential Revision: https://phab.mercurial-scm.org/D4122
Mon, 06 Aug 2018 11:32:16 -0700 changegroup: extract _revisiondeltanormal() to standalone function
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 11:32:16 -0700] rev 39023
changegroup: extract _revisiondeltanormal() to standalone function It wasn't accessing anything important on the cgpacker that warranted it being a method instead of a function. Differential Revision: https://phab.mercurial-scm.org/D4142
Mon, 06 Aug 2018 11:13:25 -0700 changegroup: inline _revchunk() into group()
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 11:13:25 -0700] rev 39022
changegroup: inline _revchunk() into group() _revchunk() was pretty minimal. I think having all the code for generating data composing the changegroup in one function makes things easier to understand. As part of the refactor, we now call the _revisiondelta* functions explicitly. This paves the road to refactor their argument signatures. Differential Revision: https://phab.mercurial-scm.org/D4141
Mon, 06 Aug 2018 11:06:22 -0700 changegroup: pass mfdicts properly
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 11:06:22 -0700] rev 39021
changegroup: pass mfdicts properly With the narrow code part of core, the hacky pass-argument-via- attribute-on-self can be accomplished with a regular function argument. Differential Revision: https://phab.mercurial-scm.org/D4140
Mon, 06 Aug 2018 11:33:05 -0700 changegroup: pass sorted revisions into group() (API)
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 11:33:05 -0700] rev 39020
changegroup: pass sorted revisions into group() (API) Currently, group() receives a list of nodes and calls _sortgroup() to sort them and turn them into revs. Since the sorting behavior varies depending on the type of data being transferred, I think it makes sense to perform the sorting before group() is invoked. This commit extracts _sortgroup() to a pair of standalone functions. It then moves the calling of these functions to the 3 call sites of group(). group() now receives an iterable of revs instead of nodes. Differential Revision: https://phab.mercurial-scm.org/D4139
Fri, 03 Aug 2018 18:40:41 -0700 changegroup: pull _fileheader out of cgpacker
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 03 Aug 2018 18:40:41 -0700] rev 39019
changegroup: pull _fileheader out of cgpacker It doesn't need any state from the packer. Differential Revision: https://phab.mercurial-scm.org/D4138
Mon, 06 Aug 2018 09:26:02 -0700 changegroup: factor changelogdone into an argument
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 09:26:02 -0700] rev 39018
changegroup: factor changelogdone into an argument The variable was basically tracking whether the current operation is being performed against the changelog or something else. So let's just pass such a flag to everything that needs to access it. I'm still not a huge fan of building changelog awareness into low-level functions like revision delta generation. But passing an argument is strictly better than state on the packer instance. Differential Revision: https://phab.mercurial-scm.org/D4137
Fri, 03 Aug 2018 18:31:00 -0700 changegroup: record changelogdone after fully consuming its data
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 03 Aug 2018 18:31:00 -0700] rev 39017
changegroup: record changelogdone after fully consuming its data Setting this as a side-effect of calling _close() is wonky. There's only one group for changelog data. So we can wait until after all data has been emitted before recording it. Differential Revision: https://phab.mercurial-scm.org/D4136
Mon, 06 Aug 2018 09:24:35 -0700 changegroup: key off changelogdone
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 09:24:35 -0700] rev 39016
changegroup: key off changelogdone We use self._changelogdone for similar checks. Let's make things consistent. Differential Revision: https://phab.mercurial-scm.org/D4135
Mon, 06 Aug 2018 10:43:05 -0700 perf: call _generatechangelog() instead of group()
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 10:43:05 -0700] rev 39015
perf: call _generatechangelog() instead of group() Now that we have a separate function for generating just the changelog bits, the perf command should call it so it gets more accurate behavior. This changes the results of this command on my hg repo significantly: ! wall 1.390502 comb 1.390000 user 1.370000 sys 0.020000 (best of 8) ! wall 1.768750 comb 1.760000 user 1.760000 sys 0.000000 (best of 6) Profiling seems to reveal that ~20% of execution time is spent in progress bar accounting and printing! If we run with progress.disable=true: ! wall 1.639134 comb 1.650000 user 1.630000 sys 0.020000 (best of 7) A nice speedup. But profiling still shows a good chunk of time being spent in progress bar accounting code. The reason is that the progress bar is conditionally enabled via an argument to cgpacker.group(). The previous code in perf.py calling into group() did not enable the progress bar but _generatechangelog() always does. I think it is important for the perf* commands to capture real-world use cases. And this code always runs with an active progress bar. So the regression is acceptable. That being said, terminal printing performance can vary substantially. I don't think perf* commands should test terminal printing unless explicitly desired. So I've disabled progress bar printing in this command. Differential Revision: https://phab.mercurial-scm.org/D4134
Fri, 03 Aug 2018 17:59:56 -0700 changegroup: factor changelog chunk generation into own function
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 03 Aug 2018 17:59:56 -0700] rev 39014
changegroup: factor changelog chunk generation into own function We have separate functions for generating manifests and filelogs. Let's split changelog into its own function so things are consistent. As part of this, we refactor the code slightly. Before, the changelog linknode callback was updating state on variables inherited via a closure. Since the closure is now separate from generate(), we need to a way pass state between generate() and _generatechangelog(). The return value of _generatechangelog() is a 2-tuple where the first item is a dict containing accumulated state. We then alias some of its members into the scope of generate() to reduce code churn. I will be converting other functions to a similar pattern in future commits. Differential Revision: https://phab.mercurial-scm.org/D4133
Fri, 03 Aug 2018 14:16:14 -0700 changegroup: pass function to resolve delta parents into constructor
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 03 Aug 2018 14:16:14 -0700] rev 39013
changegroup: pass function to resolve delta parents into constructor Previously, _deltaparent() encapsulated the logic for all 3 delta parent modes of operation. The choice of delta parent is static for the lifetime of the packer and can be passed into the packer as a callable. So do that. Differential Revision: https://phab.mercurial-scm.org/D4132
Tue, 07 Aug 2018 10:24:49 -0700 changegroup: restore original behavior of _nextclrevtolocalrev
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 10:24:49 -0700] rev 39012
changegroup: restore original behavior of _nextclrevtolocalrev 0548f696795b accidentally changed the behavior of cgpacker._close(). The old behavior moved _nextclrevtolocalrev to _clrevtolocalrev only when _nextclrevtolocalrev was present and then removed _nextclrevtolocalrev. The bad behavior performed this move then cleared _clrevtolocalrev because it was the same object as _nextclrevtolocalrev. This commit restores the previous behavior. Surprisingly, no tests changed as a result of this bad logic. I'm not sure why. Differential Revision: https://phab.mercurial-scm.org/D4155
Thu, 09 Aug 2018 12:03:39 -0400 py3: whitelist another test caught by the ratchet
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 12:03:39 -0400] rev 39011
py3: whitelist another test caught by the ratchet Differential Revision: https://phab.mercurial-scm.org/D4173
Thu, 09 Aug 2018 11:56:24 -0400 debugcommands: force import of fileset in debugfileset
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 11:56:24 -0400] rev 39010
debugcommands: force import of fileset in debugfileset It looks like Python 3's lazy importer is better than Python 2's for this command, and as a result we had no symbols in the filesetlang symbol table, which resulted in some really mysterious test-fileset.t failures around withstatus optimizations. Inserting this explicit import and forcing its evaluation fixes the test failure. Differential Revision: https://phab.mercurial-scm.org/D4172
Tue, 07 Aug 2018 17:22:33 -0700 linelog: optimize replacelines
Jun Wu <quark@fb.com> [Tue, 07 Aug 2018 17:22:33 -0700] rev 39009
linelog: optimize replacelines The optimization to avoid calling `annotate` inside `replacelines` is significant for practical use patterns. Before this patch: hg perflinelogedits ! wall 6.778478 comb 6.710000 user 6.700000 sys 0.010000 (best of 3) After this patch: hg perflinelogedits ! wall 0.136573 comb 0.140000 user 0.130000 sys 0.010000 (best of 63) Differential Revision: https://phab.mercurial-scm.org/D4150
Tue, 07 Aug 2018 17:17:01 -0700 linelog: extract `len(self._program)` to a local function
Jun Wu <quark@fb.com> [Tue, 07 Aug 2018 17:17:01 -0700] rev 39008
linelog: extract `len(self._program)` to a local function This is a micro optimization prepared for following changes where `len(self._program)` is used in a loop. Differential Revision: https://phab.mercurial-scm.org/D4149
Mon, 06 Aug 2018 18:56:24 -0700 perf: add a command to benchmark linelog edits
Jun Wu <quark@fb.com> [Mon, 06 Aug 2018 18:56:24 -0700] rev 39007
perf: add a command to benchmark linelog edits The use pattern of creating a linelog is usually by calling "replacelines" multiple times. Add a command to benchmark it. Differential Revision: https://phab.mercurial-scm.org/D4148
Mon, 06 Aug 2018 18:56:24 -0700 linelog: update internal help text
Jun Wu <quark@fb.com> [Mon, 06 Aug 2018 18:56:24 -0700] rev 39006
linelog: update internal help text This clarifies the details asked by @martinvonz on D3990. Differential Revision: https://phab.mercurial-scm.org/D4147
Tue, 07 Aug 2018 21:15:27 -0700 fix: determine fixer tool failure by exit code instead of stderr
Danny Hooper <hooper@google.com> [Tue, 07 Aug 2018 21:15:27 -0700] rev 39005
fix: determine fixer tool failure by exit code instead of stderr This seems like the more natural thing, and it probably should have been this way to beign with. It is more flexible because it allows tools to emit diagnostic information while also modifying a file. An example would be an automatic code formatter that also prints any remaining lint issues. Differential Revision: https://phab.mercurial-scm.org/D4158
Thu, 09 Aug 2018 13:13:09 +0300 status: advertise --abort instead of 'update -C .' to abort graft
Pulkit Goyal <pulkit@yandex-team.ru> [Thu, 09 Aug 2018 13:13:09 +0300] rev 39004
status: advertise --abort instead of 'update -C .' to abort graft Recent release got us a --abort flag for 'hg graft' command which is nice UI and we should advertise that to stop the graft instead of 'update -C .' which is kind of ugly. Differential Revision: https://phab.mercurial-scm.org/D4169
Thu, 09 Aug 2018 12:32:11 +0300 status: advertise --abort instead of 'update -C .' to abort a merge
Pulkit Goyal <pulkit@yandex-team.ru> [Thu, 09 Aug 2018 12:32:11 +0300] rev 39003
status: advertise --abort instead of 'update -C .' to abort a merge status has a part where it shows the conflict information and how to continue or abort. Couple of release ago, we got merge --abort and we should advertise that instead of 'hg update -C .' which is kind of ugly. I know we need to unify the logic here. Differential Revision: https://phab.mercurial-scm.org/D4168
Thu, 09 Aug 2018 12:20:28 +0300 narrow: add '()' to ellipsis in the revset help
Pulkit Goyal <pulkit@yandex-team.ru> [Thu, 09 Aug 2018 12:20:28 +0300] rev 39002
narrow: add '()' to ellipsis in the revset help ellipsis is a revset function and was missing () after it's name in the help text. This might confuse users as they try `hg log -r 'ellipsis'`. Differential Revision: https://phab.mercurial-scm.org/D4167
(0) -30000 -10000 -3000 -1000 -300 -100 -96 +96 +100 +300 +1000 +3000 +10000 tip