Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 02:30:14 -0400] rev 39097
util: fix signature of setsockopt in socket observer
The format string expected the result, callers all passed the result,
but we didn't actually accept the result to this function. I swear
I've looked at this code a dozen times trying to find the problem, and
it was this simple all along.
Differential Revision: https://phab.mercurial-scm.org/D4253
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 02:03:26 -0400] rev 39096
tests: port test-glog.t to Python 3 by using extracted printrevset
Differential Revision: https://phab.mercurial-scm.org/D4252
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 02:02:40 -0400] rev 39095
tests: extract printrevset extension from test-glog-beautifygraph.t
Differential Revision: https://phab.mercurial-scm.org/D4251
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 02:18:41 -0400] rev 39094
tests: add missing b'' prefix in test-check-code.t
It now passes on Python 3.
# skip-blame b prefix
Differential Revision: https://phab.mercurial-scm.org/D4250
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 02:17:50 -0400] rev 39093
contrib: have check-code look at files in latin1 instead of ascii
This way all files open. So far none of our patterns look like they'll care.
Differential Revision: https://phab.mercurial-scm.org/D4249
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 00:37:24 -0400] rev 39092
tests: port inline extension in test for beautifygraph to Python 3
test-glog-beautifygraph.t now passes on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D4247
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 00:08:06 -0400] rev 39091
beautifygraph: use slicing instead of subscripting on bytestr
Differential Revision: https://phab.mercurial-scm.org/D4246
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 00:07:45 -0400] rev 39090
beautifygraph: use sysstr for checking encoding._wide
# skip-blame just an r prefix
Differential Revision: https://phab.mercurial-scm.org/D4245
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 01:42:01 -0400] rev 39089
stringutil: have buildrepr delegate to pprint for unknown types
This stabilizes some revset reprs across Python 2 and 3.
Differential Revision: https://phab.mercurial-scm.org/D4244
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 01:41:31 -0400] rev 39088
stringutil: teach pprint about sets
This is the old (Python 2) way of printing sets. I actually prefer the
Python 3 version of the repr, but this will result in less test churn
in the short term.
Differential Revision: https://phab.mercurial-scm.org/D4243
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 00:12:55 -0400] rev 39087
parser: replace bespoke _brepr with stringutil.pprint
Differential Revision: https://phab.mercurial-scm.org/D4242
Augie Fackler <augie@google.com> [Fri, 10 Aug 2018 02:06:46 -0400] rev 39086
mq: use stringutil.pprint instead of pycompat.byterepr
Resolves a few output changes on Python 3 in test-mq.t, and gets us
two new passing tests.
Differential Revision: https://phab.mercurial-scm.org/D4248
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 13:27:05 -0400] rev 39085
tests: remove expected output of test-parseindex2.py
Everything uses unittest idioms now, so this can go away.
Differential Revision: https://phab.mercurial-scm.org/D4185
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 13:13:16 -0400] rev 39084
parsers: adjust pure-python version to mimic a3dacabd476b
This was caught by the tests, to my surprise. I'll also follow up with
a test of the index[-1] behavior so we can be sure that remains
consistent, as I think that currently has no coverage.
Differential Revision: https://phab.mercurial-scm.org/D4183
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 13:34:33 -0400] rev 39083
tests: restore Python 3 compat in test-parseindex2.py
I think this was previously getting lucky in Python 3, and would have
stacktraced if something failed. Now that failure messages are eagerly
computed, this turned into a problem.
Differential Revision: https://phab.mercurial-scm.org/D4182
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 13:26:28 -0400] rev 39082
tests: port remaining bits of test-parseindex2 to unittest asserts
Differential Revision: https://phab.mercurial-scm.org/D4181
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 23:13:56 -0400] rev 39081
tests: allow more detailed error message from re.compile
test-grep.t now passes on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D4240
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 23:13:35 -0400] rev 39080
grep: include format string in columns list
This lets us correctly use %d as needed.
Differential Revision: https://phab.mercurial-scm.org/D4239
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 23:13:01 -0400] rev 39079
grep: coerce username to bytestr, not str
Differential Revision: https://phab.mercurial-scm.org/D4238
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 23:12:44 -0400] rev 39078
grep: difflib sequencematcher opcodes are native strs
Differential Revision: https://phab.mercurial-scm.org/D4237
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 21:59:32 -0400] rev 39077
tests: allow for a bonus newline in base64'd email payload
Python 3 adds this newline, which is harmless.
test-patchbomb.t now passes on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D4234
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 21:58:43 -0400] rev 39076
mail: be more cautious about bytes vs str for py3 compat
It's suboptimal that we get a bytes on 2 and a unicode on 3, but it's
easy to work with and I'm disinclined to change anything unless we
start using some sort of type inferencer.
Differential Revision: https://phab.mercurial-scm.org/D4233
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 21:57:46 -0400] rev 39075
patchbomb: use native strings when determining attachment disposition
Differential Revision: https://phab.mercurial-scm.org/D4232
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 21:04:15 -0400] rev 39074
mail: always fall back to iso-8859-1 if us-ascii won't work (BC)
It looks like this was a well-intentioned backwards compat hack for
previewing the output of `hg email` in a stable way. Unfortunately I
think this hack's time has come, because Python 3 does a much better
job of ensuring it actually emits *valid* email messages. In
particular, Python 2 would blindly trust us that the bytes we handed
it were valid for the encoding we claimed, but Python 3 has some more
sniff-tests that we end up failing.
As a result, if we're going to print an email to the terminal, try
us-ascii first, but if that fails go straight to iso-8859-1 which
should be reasonably readable for ascii-compatible patch bodies. This
*will* be a breaking change for ascii-incompatible textual patch
content, but I don't think that's avoidable if we want to continue
using the email library from the stdlib.
.. bc::
Emails from the patchbomb extension will always be printed as though
they are iso-8859-1 if they're not valid us-ascii. Previously,
previewed emails were always claimed to be us-ascii and might
contain invalid byte sequences.
Differential Revision: https://phab.mercurial-scm.org/D4231
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 20:57:27 -0400] rev 39073
tests: put some Python 3 polish on inline Python invocations
A couple of these became inline python sessions because they got more
involved, and one of them is super-annoying and writes directly to
sys.stdout.buffer because I just couldn't make it work any other way.
Differential Revision: https://phab.mercurial-scm.org/D4230
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 19:39:33 -0400] rev 39072
patchbomb: don't unintentionally duplicate headers
This dict-like of headers is only mostly dict-like: if you set a key
that already exists, you it appends another value, rather than
replacing the one that was already present.
This fixes test-patchbomb-bookmark.t on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D4229
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 19:37:57 -0400] rev 39071
patchbomb: use sysstrs when describing content-type
Differential Revision: https://phab.mercurial-scm.org/D4228
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 18:28:43 -0400] rev 39070
patchbomb: use native strs for email header keys and values
Differential Revision: https://phab.mercurial-scm.org/D4223
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 18:28:19 -0400] rev 39069
patchbomb: use email.encoders instead of email.Encoders
No idea when this became a thing, but it exists for me in both 2.7 and 3.6.
Differential Revision: https://phab.mercurial-scm.org/D4222
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 18:27:36 -0400] rev 39068
mail: properly handle email addresses typically being unicodes
Visible progress in test-patchbomb.t on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D4221
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 20:08:30 -0700] rev 39067
tests: fix Python3 issues in Python one-liners in test-patchbomb.t
# skip-blame just byte prefixes
Differential Revision: https://phab.mercurial-scm.org/D4220
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 17:46:07 -0400] rev 39066
dummysmtpd: accept additional kwargs from stdlib smtpd
This was causing extremely mysterious failures because smtpd is
implemented using asynchat, which in turn doesn't appear to do
anything *remotely* helpful if an exception occurs.
# no-check-commit because I'm editing a foo_bar function signature
test-patchbomb-tls.t now passes in Python 3, but only because it
doesn't check message bodies like test-patchbomb.t. test-patchbomb.t
is *full* of doubled headers in the output (eg [0]) which seems like
an odd failure mode.
0:
@@ -141,12 +154,17 @@
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
+ MIME-Version: 1.0
+ Content-Type: text/plain; charset="us-ascii"
+ Content-Transfer-Encoding: 7bit
Subject: [PATCH] bookmark
X-Mercurial-Node: 8dab2639fd35f1e337ad866c372a5c44f1064e3c
X-Mercurial-Series-Index: 1
X-Mercurial-Series-Total: 1
Differential Revision: https://phab.mercurial-scm.org/D4219
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 17:44:36 -0400] rev 39065
mail: fix debug print, which appears to have been broken for a long time
I noticed this when trying to debug very mysterious dummysmtpd
problems. It turns out you can set self.debuglevel to a number greater
than 0 and have smtplib print helpful debug output, but this output on
our side of things was broken.
Differential Revision: https://phab.mercurial-scm.org/D4218
Augie Fackler <augie@google.com> [Mon, 16 Jul 2018 18:17:44 -0400] rev 39064
patchbomb: python 3 really wants those email addresses in unicode
At this point, test-patchbomb-tls.t almost passes on Python 3, but I'm
not really sure what the issue is: we just get a `abort: Connection
unexpectedly closed` that doesn't make sense to me.
Differential Revision: https://phab.mercurial-scm.org/D3957
Augie Fackler <augie@google.com> [Mon, 16 Jul 2018 18:17:19 -0400] rev 39063
mail: stop using the smtplib.SSLFakeFile and use socket.socket.makefile
They're equivalent, and the latter is what Python 3.3 says to use in
the release notes. Turns out it works on Python 2 as well.
Differential Revision: https://phab.mercurial-scm.org/D3956
Augie Fackler <augie@google.com> [Mon, 16 Jul 2018 18:16:26 -0400] rev 39062
mail: modernize check for Python-with-TLS
We used to be going indirectly through the socket module, but now we
just check for the ssl module.
Differential Revision: https://phab.mercurial-scm.org/D3955
Augie Fackler <augie@google.com> [Mon, 16 Jul 2018 17:49:17 -0400] rev 39061
mail: cope with Py3 unicode antics on email addresses
Differential Revision: https://phab.mercurial-scm.org/D3954
Augie Fackler <augie@google.com> [Mon, 16 Jul 2018 17:48:03 -0400] rev 39060
mail: fix _encode to be more correct on Python 3
This code appears to be on the wrong side of the law in Python 2, at
least some of the time. In Python 3, it's definitely wrong in places,
but fortunately that's easy to fix.
Differential Revision: https://phab.mercurial-scm.org/D3953
Augie Fackler <augie@google.com> [Mon, 16 Jul 2018 14:14:27 -0400] rev 39059
patchbomb: work around email module really wanting to write unicode data
Differential Revision: https://phab.mercurial-scm.org/D3951
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 09 Aug 2018 12:25:06 -0700] rev 39058
changegroup: invert conditional and dedent
I don't like else-less blocks that are indented for no reason.
Differential Revision: https://phab.mercurial-scm.org/D4217
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 09 Aug 2018 09:40:29 -0700] rev 39057
changegroup: capture base node instead of rev in delta request
I want the API in storage to be defined in terms of nodes, not
revision numbers.
Differential Revision: https://phab.mercurial-scm.org/D4216
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 09 Aug 2018 09:28:26 -0700] rev 39056
changegroup: introduce requests to define delta generation
Currently, we iterate through each revision we will be producing
a delta for then call into 1 of 2 functions for generating that
delta. Deltas are emitted as we iterate.
A problem with this model is that revision generation is tightly
coupled to the changegroup code. And the storage layer needs to
expose APIs like deltaparent() so changegroup delta generation
can produce a delta with that knowledge.
Another problem is that in this model, deltas can only be produced
sequentially after the previous delta was produced and emitted.
Some storage backends might be capable of producing deltas in
parallel (e.g. if the changegroup deltas are cached somewhere).
This commit aims to solve these problems by turning delta generation
into a 2 phase implementation where the first phase determines
info about all the deltas that need to be generated and the 2nd
phase resolves those deltas.
We introduce a "revisiondeltarequest" object that holds data about
a to-be-generated delta. We perform a full pass over all revisions
whose delta is to be generated and generate a "revisiondeltarequest"
for each. Then we iterate over the "revisiondeltarequest" instances
and derive a "revisiondelta" for each.
This patch was quite large. In order to avoid even more churn, aspects
of the implementation are less than ideal. e.g. we're recording
revision numbers instead of nodes in a few places and we don't
yet have a formal API for resolving an iterable of revisiondeltarequest
instances. Things will be improved in subsequent commits.
Unfortunately, this commit reduces performance substantially. For
`hg perfchangegroupchangelog` on my hg repo:
! wall 1.512607 comb 1.510000 user 1.490000 sys 0.020000 (best of 7)
! wall 2.150863 comb 2.150000 user 2.150000 sys 0.000000 (best of 5)
And for `hg bundle -t none-v2 -a` for the mozilla-unified repo:
178.32user 4.22system 3:02.59elapsed
190.97user 4.17system 3:15.19elapsed
Some of this was attributed to changelog slowdown. `hg
perfchangegroupchangelog` on mozilla-unified:
! wall 21.688715 comb 21.690000 user 21.570000 sys 0.120000 (best of 3)
! wall 25.683659 comb 25.680000 user 25.540000 sys 0.140000 (best of 3)
Profiling seems to reveal that the changelog slowdown is due to reading
changelog revisions multiple times. First in the linknode callback
to resolve the set of files changed. Second in the delta generation.
Before, we likely had hit the last revision cache in the revlog when
doing delta generation since we performed that immediately after
performing the linknode callback.
I'm not exactly sure where the other ~8s are being spent. It might be
from overhead of constructing a few million revisiondeltarequest
objects. I'm OK with the regression for now because it is in service
of a larger cause (storage abstraction). I'll try to profile later
and claw back the performance.
Differential Revision: https://phab.mercurial-scm.org/D4215
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 20:17:48 -0700] rev 39055
changegroup: refactor delta parent code
We had recently abstracted the delta parent functions to
facilitate extracting code from cgpacker. Now that we're in
a better place, it is time to revisit the design.
Changegroup version 1 requires that the previous node be used as the
delta parent. Later versions allow any available node to be used
as the base.
In the case where an arbitrary parent can be used, the choice of
a delta parent is best left in the hands of the storage backend.
So it makes sense for the delta parent selection to be hidden
away in the storage layer. This means deferring the choice of the
delta parent selection function to as close to delta generation
time as possible.
This commit moves the delta selection logic to essentially just
before delta generation. However, because changegroup version 1
limits what we can do, we have retained the ability to force a
delta against the previous revision.
As part of this, I realized that the ellipsis parent function was
unused! That's because ellipsis mode always sends full revisions
and not deltas.
Differential Revision: https://phab.mercurial-scm.org/D4214
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 16:01:26 -0700] rev 39054
changegroup: differentiate between fulltext and diff based deltas
Previously, revisiondelta encoded a delta and an optional prefix
containing a delta header. The underlying code could populate
the delta with either a real delta or a fulltext revision.
Following the theme of wanting to defer serialization of revision
data to the changegroup format as long as possible, it seems
prudent for the revision delta instance to capture what type of
data is being represented. This could possibly allow us to
encode revision data differently in the future. But for the
short term, it makes the behavior of a revisiondelta more
explicit.
Differential Revision: https://phab.mercurial-scm.org/D4213
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 15:28:22 -0700] rev 39053
changegroup: minor cleanups to deltagroup()
Differential Revision: https://phab.mercurial-scm.org/D4212
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 14:44:48 -0700] rev 39052
changegroup: emit revisiondelta instances from deltagroup()
By abstracting the concept of a delta group away from its
serialization (the changegroup format), we make it easier
to establish alternate serialization formats. We also make
it possible to move aspects of delta group generation into
the storage layer. This will allow storage to make decisions
about e.g. delta parent choices without the changegroup code
needing storage APIs to determine delta parents. We're still
a bit of a way from there. Future commits will work towards
that world.
Differential Revision: https://phab.mercurial-scm.org/D4211
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 14:33:33 -0700] rev 39051
changegroup: move file chunk emission to generate()
Same deal as manifests. We want to get to a point where we can
emit data structures from deltagroup() and derive the raw
changegroup data as late as possible.
Differential Revision: https://phab.mercurial-scm.org/D4210
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 15:14:59 -0700] rev 39050
changegroup: move manifest chunk emission to generate()
We want to get to a point where we can emit data structures from
deltagroup() and derive the raw changegroup data as late as possible.
Differential Revision: https://phab.mercurial-scm.org/D4209
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 15:09:12 -0700] rev 39049
changegroup: move size tracking and end of manifests to generate()
Preparing for all the generate* functions to emit data structures
instead of raw chunks.
Differential Revision: https://phab.mercurial-scm.org/D4208
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 14:15:33 -0700] rev 39048
changegroup: emit delta group close chunk outside of deltagroup()
I want to make deltagroup() emit data structures rather than
serialized deltas. Upcoming commits will demonstrate why.
Differential Revision: https://phab.mercurial-scm.org/D4207
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 14:19:02 -0700] rev 39047
changegroup: extract cgpacker.group() to standalone function
It doesn't need to be part of the packer class.
Differential Revision: https://phab.mercurial-scm.org/D4206
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 14:02:31 -0700] rev 39046
changegroup: pass all state into group()
This will allow us to split it into a standalone function.
Differential Revision: https://phab.mercurial-scm.org/D4205
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 08 Aug 2018 13:50:54 -0700] rev 39045
changegroup: inline _prune() into call sites
The functionality is pretty simple. As a bonus, _prune() had special
code for the manifest case. We can now exclude this check from the
file call site.
Differential Revision: https://phab.mercurial-scm.org/D4199
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 15:31:03 -0700] rev 39044
changegroup: inline _packmanifests() into generatemanifests()
It is relatively small. Every other generate*() calls group()
directly. So the new code is consistent.
Differential Revision: https://phab.mercurial-scm.org/D4198
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 15:13:25 -0700] rev 39043
changegroup: invert conditional and dedent
I don't like seeing code that visually resembles the pyramid of
doom.
Differential Revision: https://phab.mercurial-scm.org/D4197
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 15:10:38 -0700] rev 39042
changegroup: make _revisiondeltanarrow() a standalone function
It doesn't require any state on the packer. Everything impacting
behavior is passed in as a function. So split it out, just like
what was done for _revisiondeltanormal().
Differential Revision: https://phab.mercurial-scm.org/D4196
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 15:08:29 -0700] rev 39041
changegroup: pass state into _revisiondeltanarrow
After this, the method no longer accesses self and can be split
into a standalone function.
Differential Revision: https://phab.mercurial-scm.org/D4195
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 14:53:42 -0700] rev 39040
changegroup: inline _close()
Now that it doesn't clear self._clrevtolocalrev on every invocation
and is a simple one-liner that calls another function, we can
do away with this method and inline its content into all call
sites.
Differential Revision: https://phab.mercurial-scm.org/D4194
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 14:52:00 -0700] rev 39039
changegroup: pass clrevtolocalrev to each group
clrevtolocalrev is a per-changegroup group mapping revisions to
aid with shallow clone.
Back when this functionality was implemented in an extension, this
dict was added to the packer instance so monkeypatched functions
could reference it there. Now that this code is part of core, we
can pass the dict to each consumer properly so it doesn't have to
live on the cgpacker instance. This commit does that.
Differential Revision: https://phab.mercurial-scm.org/D4193
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 12:44:56 -0700] rev 39038
changegroup: combine _generatefiles() into generatefiles()
These were split out in a06aab274aef as part of moving the
narrow code into core. They don't need to be separate
functions.
Differential Revision: https://phab.mercurial-scm.org/D4192
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 15:45:56 -0700] rev 39037
changegroup: define linknodes callbacks in generatefiles()
This is how it is done everywhere else.
But the logic here is a bit more complex because shallow clone
needs to reference the original linknode implementation. But
at least now all function implementations are defined in the
same place.
Differential Revision: https://phab.mercurial-scm.org/D4191
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 10:55:32 -0700] rev 39036
changegroup: track changelog to manifest revision map explicitly
Previously, self._nextclrevtolocalrev was only populated as part
of the changelog lookup callback. But cgpacker._close() was looking
at self._nextclrevtolocalrev on every invocation.
Since self._nextclrevtolocalrev is for communicating the mapping
of changelog revisions to manifest revisions, this commit refactors
the code to make that explicit.
The changelog state now stores this mapping. And after the changelog
group is emitted, we update self._clrevtolocalrev with that dict.
self._nextclrevtolocalrev is unused and has been deleted.
Differential Revision: https://phab.mercurial-scm.org/D4190
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 10:49:41 -0700] rev 39035
changegroup: remove _clnodetorev
cgpacker._clnodetorev is a glorified cache/index of changelog
nodes to revision numbers.
I'm not sure why it exists. Maybe performance? But its presence
is making refactoring of this code more complicated than it needs
to be.
This commit removes the cache and replaces it with direct lookups
against the changelog.
If this cache was for performance reasons, we should be able to
restore it easily enough... after the changegroup refactor is
complete.
Differential Revision: https://phab.mercurial-scm.org/D4189
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 15:44:33 -0700] rev 39034
changegroup: rename _fullnodes to _fullclnodes
So it is obvious which nodes we are talking about.
And sneak in a docs change to reflect that this variable is a set.
Differential Revision: https://phab.mercurial-scm.org/D4188
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 15:04:20 -0700] rev 39033
changegroup: move part of _revisiondeltanarrow into group()
Now all the logic for determining which delta generation code
is called lives in a single function.
Differential Revision: https://phab.mercurial-scm.org/D4187
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 14:56:37 -0700] rev 39032
changegroup: populate _clnodetorev as part of changelog linknode lookup
The thing that matters is that self._clnodetorev is populated with
changesets that are being sent. Back when this code was in an
extension, it wasn't possible to monkeypatch the changelog lookup
function. Now that the code is in core, we can move this code to
where it logically belongs.
Differential Revision: https://phab.mercurial-scm.org/D4186
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 13:08:29 -0400] rev 39031
tests: rename variables in revlog index parse test for clarity
Now it's unambiguous which one is the expected value. c_res_{1,2} was
also misleading a bit because in --pure mode we're testing the old
slow Python version against the modern optimized Python version.
Differential Revision: https://phab.mercurial-scm.org/D4180
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 13:06:50 -0400] rev 39030
tests: move assertion closer to want/got declarations in test-parseindex2.py
I find this easier to understand.
Differential Revision: https://phab.mercurial-scm.org/D4179
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 13:05:40 -0400] rev 39029
tests: move chunks of test-parseindex2.py to use unittest properly
This doesn't touch the version-detection tests yet, because those are
more involved.
Differential Revision: https://phab.mercurial-scm.org/D4178
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 12:59:23 -0400] rev 39028
tests: fix up indent width in test-parseindex2.py
Differential Revision: https://phab.mercurial-scm.org/D4177
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 12:58:25 -0400] rev 39027
tests: start moving test-parseindex2.py to a unittest
Using 2-space indents in this revision to make the code motion easier
to review. I'll fix it in the next commit.
Differential Revision: https://phab.mercurial-scm.org/D4176
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 12:10:34 -0400] rev 39026
tests: port test-absorb-filefixupstate to Python 3
Mostly b prefixes, but also some isinstance() checks and a couple of
maplist() instances. The test now passes on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D4175
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 12:06:31 -0400] rev 39025
absorb: port partway to Python 3
Use pycompat.maplist() in the one place that matters and use the
default iterator of a dict instead of iterkeys().
Two new tests pass on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D4174
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Sat, 04 Aug 2018 21:31:46 -0400] rev 39024
localrepo: better error when a repo exists but we lack permissions
Claiming "repository foo not found" when the repository does exist
causes confusion regularly ("where is the typo?").
Differential Revision: https://phab.mercurial-scm.org/D4122
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 11:32:16 -0700] rev 39023
changegroup: extract _revisiondeltanormal() to standalone function
It wasn't accessing anything important on the cgpacker that warranted
it being a method instead of a function.
Differential Revision: https://phab.mercurial-scm.org/D4142
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 11:13:25 -0700] rev 39022
changegroup: inline _revchunk() into group()
_revchunk() was pretty minimal. I think having all the code for
generating data composing the changegroup in one function makes
things easier to understand.
As part of the refactor, we now call the _revisiondelta* functions
explicitly. This paves the road to refactor their argument
signatures.
Differential Revision: https://phab.mercurial-scm.org/D4141
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 11:06:22 -0700] rev 39021
changegroup: pass mfdicts properly
With the narrow code part of core, the hacky pass-argument-via-
attribute-on-self can be accomplished with a regular function
argument.
Differential Revision: https://phab.mercurial-scm.org/D4140
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 11:33:05 -0700] rev 39020
changegroup: pass sorted revisions into group() (API)
Currently, group() receives a list of nodes and calls _sortgroup()
to sort them and turn them into revs. Since the sorting behavior
varies depending on the type of data being transferred, I think it
makes sense to perform the sorting before group() is invoked.
This commit extracts _sortgroup() to a pair of standalone functions.
It then moves the calling of these functions to the 3 call sites of
group(). group() now receives an iterable of revs instead of nodes.
Differential Revision: https://phab.mercurial-scm.org/D4139
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 03 Aug 2018 18:40:41 -0700] rev 39019
changegroup: pull _fileheader out of cgpacker
It doesn't need any state from the packer.
Differential Revision: https://phab.mercurial-scm.org/D4138
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 09:26:02 -0700] rev 39018
changegroup: factor changelogdone into an argument
The variable was basically tracking whether the current operation
is being performed against the changelog or something else. So
let's just pass such a flag to everything that needs to access it.
I'm still not a huge fan of building changelog awareness into
low-level functions like revision delta generation. But passing
an argument is strictly better than state on the packer instance.
Differential Revision: https://phab.mercurial-scm.org/D4137
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 03 Aug 2018 18:31:00 -0700] rev 39017
changegroup: record changelogdone after fully consuming its data
Setting this as a side-effect of calling _close() is wonky. There's
only one group for changelog data. So we can wait until after all
data has been emitted before recording it.
Differential Revision: https://phab.mercurial-scm.org/D4136
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 09:24:35 -0700] rev 39016
changegroup: key off changelogdone
We use self._changelogdone for similar checks. Let's make things
consistent.
Differential Revision: https://phab.mercurial-scm.org/D4135
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 06 Aug 2018 10:43:05 -0700] rev 39015
perf: call _generatechangelog() instead of group()
Now that we have a separate function for generating just the changelog
bits, the perf command should call it so it gets more accurate
behavior.
This changes the results of this command on my hg repo significantly:
! wall 1.390502 comb 1.390000 user 1.370000 sys 0.020000 (best of 8)
! wall 1.768750 comb 1.760000 user 1.760000 sys 0.000000 (best of 6)
Profiling seems to reveal that ~20% of execution time is spent in
progress bar accounting and printing! If we run with
progress.disable=true:
! wall 1.639134 comb 1.650000 user 1.630000 sys 0.020000 (best of 7)
A nice speedup. But profiling still shows a good chunk of time being
spent in progress bar accounting code. The reason is that the
progress bar is conditionally enabled via an argument to
cgpacker.group(). The previous code in perf.py calling into group()
did not enable the progress bar but _generatechangelog() always does.
I think it is important for the perf* commands to capture real-world
use cases. And this code always runs with an active progress bar. So
the regression is acceptable.
That being said, terminal printing performance can vary substantially.
I don't think perf* commands should test terminal printing unless
explicitly desired. So I've disabled progress bar printing in this
command.
Differential Revision: https://phab.mercurial-scm.org/D4134
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 03 Aug 2018 17:59:56 -0700] rev 39014
changegroup: factor changelog chunk generation into own function
We have separate functions for generating manifests and filelogs.
Let's split changelog into its own function so things are consistent.
As part of this, we refactor the code slightly. Before, the
changelog linknode callback was updating state on variables
inherited via a closure. Since the closure is now separate from
generate(), we need to a way pass state between generate() and
_generatechangelog(). The return value of _generatechangelog()
is a 2-tuple where the first item is a dict containing accumulated
state. We then alias some of its members into the scope of
generate() to reduce code churn.
I will be converting other functions to a similar pattern in future
commits.
Differential Revision: https://phab.mercurial-scm.org/D4133
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 03 Aug 2018 14:16:14 -0700] rev 39013
changegroup: pass function to resolve delta parents into constructor
Previously, _deltaparent() encapsulated the logic for all 3
delta parent modes of operation. The choice of delta parent
is static for the lifetime of the packer and can be passed into
the packer as a callable. So do that.
Differential Revision: https://phab.mercurial-scm.org/D4132
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Aug 2018 10:24:49 -0700] rev 39012
changegroup: restore original behavior of _nextclrevtolocalrev
0548f696795b accidentally changed the behavior of cgpacker._close().
The old behavior moved _nextclrevtolocalrev to _clrevtolocalrev only
when _nextclrevtolocalrev was present and then removed
_nextclrevtolocalrev. The bad behavior performed this move
then cleared _clrevtolocalrev because it was the same object as
_nextclrevtolocalrev.
This commit restores the previous behavior.
Surprisingly, no tests changed as a result of this bad logic. I'm
not sure why.
Differential Revision: https://phab.mercurial-scm.org/D4155
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 12:03:39 -0400] rev 39011
py3: whitelist another test caught by the ratchet
Differential Revision: https://phab.mercurial-scm.org/D4173
Augie Fackler <augie@google.com> [Thu, 09 Aug 2018 11:56:24 -0400] rev 39010
debugcommands: force import of fileset in debugfileset
It looks like Python 3's lazy importer is better than Python 2's for
this command, and as a result we had no symbols in the filesetlang
symbol table, which resulted in some really mysterious test-fileset.t
failures around withstatus optimizations. Inserting this explicit
import and forcing its evaluation fixes the test failure.
Differential Revision: https://phab.mercurial-scm.org/D4172
Jun Wu <quark@fb.com> [Tue, 07 Aug 2018 17:22:33 -0700] rev 39009
linelog: optimize replacelines
The optimization to avoid calling `annotate` inside `replacelines` is significant
for practical use patterns.
Before this patch:
hg perflinelogedits
! wall 6.778478 comb 6.710000 user 6.700000 sys 0.010000 (best of 3)
After this patch:
hg perflinelogedits
! wall 0.136573 comb 0.140000 user 0.130000 sys 0.010000 (best of 63)
Differential Revision: https://phab.mercurial-scm.org/D4150
Jun Wu <quark@fb.com> [Tue, 07 Aug 2018 17:17:01 -0700] rev 39008
linelog: extract `len(self._program)` to a local function
This is a micro optimization prepared for following changes where
`len(self._program)` is used in a loop.
Differential Revision: https://phab.mercurial-scm.org/D4149
Jun Wu <quark@fb.com> [Mon, 06 Aug 2018 18:56:24 -0700] rev 39007
perf: add a command to benchmark linelog edits
The use pattern of creating a linelog is usually by calling "replacelines"
multiple times. Add a command to benchmark it.
Differential Revision: https://phab.mercurial-scm.org/D4148
Jun Wu <quark@fb.com> [Mon, 06 Aug 2018 18:56:24 -0700] rev 39006
linelog: update internal help text
This clarifies the details asked by @martinvonz on D3990.
Differential Revision: https://phab.mercurial-scm.org/D4147
Danny Hooper <hooper@google.com> [Tue, 07 Aug 2018 21:15:27 -0700] rev 39005
fix: determine fixer tool failure by exit code instead of stderr
This seems like the more natural thing, and it probably should have been this
way to beign with. It is more flexible because it allows tools to emit
diagnostic information while also modifying a file. An example would be an
automatic code formatter that also prints any remaining lint issues.
Differential Revision: https://phab.mercurial-scm.org/D4158
Pulkit Goyal <pulkit@yandex-team.ru> [Thu, 09 Aug 2018 13:13:09 +0300] rev 39004
status: advertise --abort instead of 'update -C .' to abort graft
Recent release got us a --abort flag for 'hg graft' command which is nice UI and
we should advertise that to stop the graft instead of 'update -C .' which is
kind of ugly.
Differential Revision: https://phab.mercurial-scm.org/D4169
Pulkit Goyal <pulkit@yandex-team.ru> [Thu, 09 Aug 2018 12:32:11 +0300] rev 39003
status: advertise --abort instead of 'update -C .' to abort a merge
status has a part where it shows the conflict information and how to continue or
abort. Couple of release ago, we got merge --abort and we should advertise that
instead of 'hg update -C .' which is kind of ugly.
I know we need to unify the logic here.
Differential Revision: https://phab.mercurial-scm.org/D4168
Pulkit Goyal <pulkit@yandex-team.ru> [Thu, 09 Aug 2018 12:20:28 +0300] rev 39002
narrow: add '()' to ellipsis in the revset help
ellipsis is a revset function and was missing () after it's name in the help
text. This might confuse users as they try `hg log -r 'ellipsis'`.
Differential Revision: https://phab.mercurial-scm.org/D4167