Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 10 Oct 2019 17:18:46 +0200] rev 43199
copies: drop the findlimit logic
We don't use the limit anymore so we should stop computing that limit.
I did not bother measuring the potential performance gain. I am assuming that
not running any code will be faster that doing some computation and not using
the result.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 10 Oct 2019 03:49:33 +0200] rev 43198
pathcopies: give up any optimization based on `introrev`
Between 8a0136f69027 and d98fb3f42f33, we sped up the search for the
introduction revision during path copies. However, further checking show that
finding the introduction revision is still expensive and that we are better off
without it. So we simply drop it and only rely on the linkrev optimisation.
I ran `perfpathcopies` on 6989 pair of revision in the pypy
repository (`hg perfhelper-pathcopies`. The result is massively in favor of
dropping this condition. The result of the copy tracing are unchanged.
Attempt to use a smaller changes preserving linkrev usage were unsuccessful, it
can return wrong result. The following changesets broke test-mv-cp-st-diff.t
- if not f.isintroducedafter(limit):
+ if limit >= 0 and f.linkrev() < limit:
return None
Here are various numbers (before this changeset/after this changesets)
source destination before after saved-time ratio
worth cases e66f24650daf 695dfb0f493b 1.062843 1.246369 -0.183526 1.172675
c979853a3b6a 8d60fe293e79 1.036985 1.196414 -0.159429 1.153743
22349fa2fc33 fbb1c9fd86c0 0.879926 1.038682 -0.158756 1.180420
682b98f3e672 a4878080a536 0.909952 1.063801 -0.153849 1.169074
5adabc9b9848 920958a93997 0.993622 1.147452 -0.153830 1.154817
worse 1% dbfbfcf077e9 aea8f2fd3593 1.016595 1.082999 -0.066404 1.065320
worse 5% c95f1ced15f2 7d29d5e39734 0.453694 0.471156 -0.017462 1.038488
worse 10% 3e144ed1d5b7 2aef0e942480 0.035140 0.037535 -0.002395 1.068156
worse 25% 321fc60db035 801748ba582a 0.009267 0.009325 -0.000058 1.006259
median 2088ce763fc2 e6991321d78b 0.000665 0.000651 0.000014 0.978947
best 25% 915631a97de6 385b31354be6 0.040743 0.040363 0.000380 0.990673
best 10% ad495c36a765 19c10384d3e7 0.431658 0.411490 0.020168 0.953278
best 5% d13ae7d283ae 813c99f810ac 1.141404 1.075346 0.066058 0.942126
best 1% 81593cb4a496 99ae11866969 1.833297 0.063823 1.769474 0.034813
best cases c3b14617fbd7 743a0fcaa4eb 1101.811740 2.735970 1099.075770 0.002483
c3b14617fbd7 9ba6ab77fd29 1116.753953 2.800729 1113.953224 0.002508
058b99d6e81f 57e249b7a3ea 1246.128485 3.042762 1243.085723 0.002442
9a8c361aab49 0354a250d371 1253.111894 3.085796 1250.026098 0.002463
442dbbc53c68 3ec1002a818c 1261.786294 3.138607 1258.647687 0.002487
As one can see, the average case is not really impacted. However, the worth case
we get after this changeset are much better than the one we had before it. We
have 30 pairs where improvements are above 10 minutes.
This reflect in the combined time for all pairs
before: 26256s
after: 1300s (-95%)
If we remove these pathological 30 cases, we still see a significant improvements:
before: 1631s
after: 1245s (-24%)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 08 Oct 2019 04:28:23 -0400] rev 43197
perf: introduce a `--contains` flag to the `perfdirstate` command
The new flag benchmark a large amount of `filepath in dirstate` call. This will
be useful to compare the Python and Rust implementation of the dirstatemap.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 08 Oct 2019 04:23:04 -0400] rev 43196
perf: introduce a `--iteration` to `perfdirstate`
This flag benchmark an iteration over all the file in the dirstate. This
will be useful to compare the Python and the Rust implementation of the
dirstate.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 13 Oct 2019 00:15:36 -0400] rev 43195
tests: open file in binary mode in test-upgrade-repo.t
Otherwise the file sizes printed in the output diverge from other platforms.
Differential Revision: https://phab.mercurial-scm.org/D7081
Matt Harbison <matt_harbison@yahoo.com> [Sat, 12 Oct 2019 22:21:51 -0400] rev 43194
hghave: use a native string to invoke the `black` command
Windows can't create a process based on bytes. Apparently only the output test
in `matchoutput` wants bytes.
Differential Revision: https://phab.mercurial-scm.org/D7080
Matt Harbison <matt_harbison@yahoo.com> [Sat, 12 Oct 2019 22:22:17 -0400] rev 43193
py3: convert cwd to native string when running `fix`
This prevents a bunch of failures on Windows.
Differential Revision: https://phab.mercurial-scm.org/D7079
Martin von Zweigbergk <martinvonz@google.com> [Fri, 11 Oct 2019 18:56:59 -0700] rev 43192
format: format commands.py, which recently regressed
# skip-blame just reformatting
Differential Revision: https://phab.mercurial-scm.org/D7064
Ian Moody <moz-ian@perix.co.uk> [Fri, 04 Oct 2019 19:36:21 +0100] rev 43191
py3: fix test-bad-extension expectations for py35
ModuleNotFoundError was introduced in 3.6, so the tests need to expect to see
ImportError with 3.5.
Differential Revision: https://phab.mercurial-scm.org/D7063
Ian Moody <moz-ian@perix.co.uk> [Sun, 06 Oct 2019 16:57:26 +0100] rev 43190
phabricator: add addremoved and addmodified functions
These are relatively simple so add them together. As the name indicates, these
add phabchange objects for removed and modified files to the phabdiff.
Differential Revision: https://phab.mercurial-scm.org/D7049
Ian Moody <moz-ian@perix.co.uk> [Sun, 06 Oct 2019 15:37:13 +0100] rev 43189
phabricator: add makebinary and addoldbinary functions
These populate the phabchange with the data for a binary file, much as maketext
does for text files.
Differential Revision: https://phab.mercurial-scm.org/D7048
Ian Moody <moz-ian@perix.co.uk> [Sun, 06 Oct 2019 15:16:47 +0100] rev 43188
phabricator: add the uploadfile function
This is needed to be able to submit binary files such as images in commits (and
also non-UTF-8 text files). One of the primary disadvantages of the current
use of createrawdiff is that Phabricator simply drops any binary diffs included
in the patch, but not the files, so if such a diff is then committed it
corrupts any binary files in it.
Differential Revision: https://phab.mercurial-scm.org/D7047
Ian Moody <moz-ian@perix.co.uk> [Sun, 06 Oct 2019 15:04:54 +0100] rev 43187
phabricator: add the uploadchunks function
Differential Revision: https://phab.mercurial-scm.org/D7046
Ian Moody <moz-ian@perix.co.uk> [Sun, 06 Oct 2019 14:54:46 +0100] rev 43186
phabricator: add the maketext function
This add the diff data for a text file to a phabchange.
Differential Revision: https://phab.mercurial-scm.org/D7045
Ian Moody <moz-ian@perix.co.uk> [Sun, 06 Oct 2019 14:53:03 +0100] rev 43185
phabricator: add the phabdiff data structure
This holds all the data about a commit, and is passed to the
differential.creatediff API.
Differential Revision: https://phab.mercurial-scm.org/D7044
Ian Moody <moz-ian@perix.co.uk> [Sun, 06 Oct 2019 14:08:03 +0100] rev 43184
phabricator: add the phabchange data structure
These store data about individual files in a commit.
Differential Revision: https://phab.mercurial-scm.org/D7043
Ian Moody <moz-ian@perix.co.uk> [Sun, 06 Oct 2019 13:55:04 +0100] rev 43183
phabricator: add the phabhunk data structure
These store the actual diff data (for UTF-8 text files anyway) and are
equivalent to hunks in a patch file.
Differential Revision: https://phab.mercurial-scm.org/D7042
Ian Moody <moz-ian@perix.co.uk> [Sun, 06 Oct 2019 13:50:32 +0100] rev 43182
phabricator: add the DiffChangeType and DiffFileType constants
These are used in Phabricator change objects.
There are more values but so far as I can tell we don't need them.
Differential Revision: https://phab.mercurial-scm.org/D7041
Rodrigo Damazio <rdamazio@google.com> [Sun, 06 Oct 2019 03:30:00 -0400] rev 43181
help: adding a help category to narrow and remotefilelog commands
Differential Revision: https://phab.mercurial-scm.org/D6998
Yuya Nishihara <yuya@tcha.org> [Sun, 15 Sep 2019 17:02:31 +0900] rev 43180
rust-cpython: drop self.borrow_mut() in favor of PySharedRef wrapper
Yuya Nishihara <yuya@tcha.org> [Sun, 15 Sep 2019 16:59:50 +0900] rev 43179
rust-cpython: drop self.leak_immutable() in favor of PySharedRef wrapper
Yuya Nishihara <yuya@tcha.org> [Sun, 15 Sep 2019 16:50:48 +0900] rev 43178
rust-cpython: add safe wrapper representing shared data borrowed from PyObject
PySharedRef is a tempoary wrapper around PySharedRefCell. It provides safe
functions for each shared data. $shared_accessor implements a safe method
to construct PySharedRefCell.
This allows us to add more than once PySharedRefCell to a Python object.
Yuya Nishihara <yuya@tcha.org> [Sun, 08 Sep 2019 20:26:55 +0900] rev 43177
rust-cpython: move $leaked struct out of macro
It wasn't easy to hack the $leaked struct since errors in macro would
generate lots of compile errors. Let's make it a plain struct so we can
easily extend it.
PyLeakedRef keeps a more generic PyObject instead of the $name struct
since it no longer has to call any specific methods implemented by
the $name class. $leaked parameter in py_shared_iterator!() is kept
for future change.
Yuya Nishihara <yuya@tcha.org> [Sun, 15 Sep 2019 16:04:45 +0900] rev 43176
rust-cpython: store leaked reference to PySharedState in $leaked struct
I want to move it out of the macro, and allow multiple sharable objects
per PyObject.
Yuya Nishihara <yuya@tcha.org> [Tue, 17 Sep 2019 07:59:25 +0900] rev 43175
rust-cpython: mark PySharedState as Sync so &'PySharedState can be Send
The goal is to store &'static PySharedState in $leaked struct, which allows
us to move the $leaked struct out of the macro. Currently, it depends on
inner.$data_member(py), which can't be generalized.
PySharedState is Sync because any mutation or read operation is synchronized
by the Python GIL, py: Python<'a>, which should guarantee that &'PySharedState
can be sent to another thread.
Yuya Nishihara <yuya@tcha.org> [Sat, 14 Sep 2019 23:17:19 +0900] rev 43174
rust-cpython: move borrow_mut() to PySharedRefCell
PySharedRefCell() will host almost all py_shared public functions. This change
is the first step.
borrow_mut() can be safely implemented since PySharedRefCell knows its inner
object is managed by its own py_shared_state.
Yuya Nishihara <yuya@tcha.org> [Sat, 14 Sep 2019 23:01:51 +0900] rev 43173
rust-cpython: move py_shared_state to PySharedRefCell object
The goal of this series is to encapsulate more "py_shared" thingy and
reduce the size of the macro, which is hard to debug.
Since py_shared_state manages the borrowing state of the object owned by
PySharedRefCell, this change makes more sense. If a PyObject has more than
one data to be leaked into Python world, each PySharedState should incref
the parent PyObject, and keep track of the corresponding borrowing state.
Denis Laxalde <denis@laxalde.org> [Thu, 10 Oct 2019 21:37:12 +0200] rev 43172
py3: decode username and password before SMTP login
smtplib.SMTP.login() requires str on Python 3.
For 'password', we only need to decode when value comes from config as
getpass() returns the correct type already.
Denis Laxalde <denis@laxalde.org> [Thu, 10 Oct 2019 21:30:44 +0200] rev 43171
py3: use socket.makefile() instead of dropped smtplib.SSLFakeFile
The latter (undocumented internal) class got removed in Python 3.3. Use
socket.makefile() as suggested:
https://docs.python.org/3/whatsnew/3.3.html#porting-python-code
Denis Laxalde <denis@laxalde.org> [Thu, 10 Oct 2019 21:05:34 +0200] rev 43170
py3: call SMTP.docmd() with an str
Otherwise, this always returns (502, '5.5.2 Error: command not
recognized').
Denis Laxalde <denis@laxalde.org> [Thu, 10 Oct 2019 21:00:13 +0200] rev 43169
py3: call SMTP.has_extn() with an str
Passing a bytes on Python 3 always returns False, thus starttls is not
properly detected.
Denis Laxalde <denis@laxalde.org> [Thu, 10 Oct 2019 20:27:34 +0200] rev 43168
py3: fix sorting of obsolete markers when building bundle
Last item of marker tuple (parents) is either None or tuple. Comparison
thus fails on Python 3 with:
TypeError: '<' not supported between instances of 'tuple' and 'NoneType'
Fixing this by coercing None to the empty tuple when sorting markers in
exchange._getbundleobsmarkerpart().
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 10 Oct 2019 04:34:58 +0200] rev 43167
sidedata: rename the configuration option to `exp-use-side-data`
We don't want setup configured to use the final version that would end using the
experimental one while using and older version.
Differential Revision: https://phab.mercurial-scm.org/D7040
Denis Laxalde <denis.laxalde@logilab.fr> [Thu, 10 Oct 2019 16:02:47 +0200] rev 43166
py3: drop ui.flush() during interactive patch filtering
With previous changeset fixing line buffering on stdout, this is no
longer needed.
Denis Laxalde <denis.laxalde@logilab.fr> [Thu, 10 Oct 2019 16:00:02 +0200] rev 43165
py3: keep stdout as defined by pycompat in procutil
According to https://docs.python.org/3/library/functions.html#open, it's
not possible to use 1 as buffering argument value in binary mode. This
is probably why line buffering does not work well in python3.
On the other hand, sys.stdout.buffer appears to be line-buffered already
on python3. So by not replacing it, there should be no behavior change.
This fixes buffering issue in "hg email" (confirmation prompt shown
before information to be confirmed).
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Oct 2019 21:21:16 -0400] rev 43164
notify: cast hash to bytes
This is needed to avoid a str/bytes mismatch when interpolating a
line or 2 later.
Differential Revision: https://phab.mercurial-scm.org/D7021
Kyle Lippincott <spectral@google.com> [Sat, 05 Oct 2019 13:39:35 -0700] rev 43163
push: support config option to require revs be specified when running push
Differential Revision: https://phab.mercurial-scm.org/D6989
Georges Racinet <georges.racinet@octobus.net> [Thu, 10 Oct 2019 11:33:33 +0200] rev 43162
py3-discovery: using plain str in stats dict
rust-cpython converts automatically from Rust strings to
the appropriate `str` for the target Python version.
Insisting on discovery stats dict keys to be bytes hence breaks
the process (this is spotted by test-setdiscovery.t).
Now that byteify-strings has been run on the entire
codebase, and the import transformer is not there any more,
the simplest fix is to make the keys plain str again.
Another possible fix would be to forcefully convert to bytes in
rust-cpython code, but that feels less natural, and would probably
have to be reverted down the road.
Differential Revision: https://phab.mercurial-scm.org/D7039
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 10 Oct 2019 04:48:31 +0200] rev 43161
perf: fix `perfhelper-mergecopies` report of #changesets
Same as `perfhelper-pathcopies`. The previous computation `<base>::<target>` was
wrong, what we actually need is `::<target> - ::<base>`.
This is now fixed.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 02 Oct 2019 18:39:20 -0400] rev 43160
perf: fix `perfhelper-pathcopies` report of #changesets
The previous computation `<base>::<target>` was wrong, what we actually need is
`::<target> - ::<base>`.
This is now fixed
Denis Laxalde <denis.laxalde@logilab.fr> [Thu, 10 Oct 2019 12:22:15 +0200] rev 43159
py3: use integer division in curseschunkselector.printstring()
This fixes a crash when scrolling in curses UI when refresh() is called
when a float value (namely 'self.firstlineofpadtoprint', taking its
value indirectly from 'self.linesprintedtopadsofar').
Denis Laxalde <denis.laxalde@logilab.fr> [Thu, 10 Oct 2019 12:20:23 +0200] rev 43158
crecord: drop duplicated set of firstlineofpadtoprint attribute
The attribute is already set a couple of lines above, in
curseschunkselector.__init__().
Denis Laxalde <denis.laxalde@logilab.fr> [Thu, 10 Oct 2019 10:53:13 +0200] rev 43157
patchbomb: use mail.Generator alias for py2/py3 compat
Denis Laxalde <denis.laxalde@logilab.fr> [Thu, 10 Oct 2019 10:48:57 +0200] rev 43156
py3: use email.generator.BytesGenerator in patch.split()
This fixes test-import.t on python3.
We add Generator alias in mail module to handle python2/python3
compatibility.
Denis Laxalde <denis.laxalde@logilab.fr> [Thu, 10 Oct 2019 10:03:01 +0200] rev 43155
py3: only flush before prompting during interactive patch filtering
Follows up on c9093ae8d6c4. It's enough to flush just before each
prompt.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 28 Aug 2019 17:45:18 -0700] rev 43154
py3: add a missing b'' prefix in test extension for chg
Differential Revision: https://phab.mercurial-scm.org/D7038
Augie Fackler <augie@google.com> [Wed, 09 Oct 2019 20:49:58 -0700] rev 43153
fuzz: new target to fuzz jsonescapeu8fast
This code just feels complicated enough we should go ahead and give it
a dedicated fuzzer: we've found bugs in similar things before.
Differential Revision: https://phab.mercurial-scm.org/D7034
Augie Fackler <augie@google.com> [Wed, 09 Oct 2019 20:49:39 -0700] rev 43152
fuzz: new fuzzer for fncache-related functions
Not all of these are strictly fncache-related, but they all have th
same signature and similar-enough behavior that we may as well fuzz
them together. No obvious bugs for once, but these felt like they were
just complicated enough to cover.
Differential Revision: https://phab.mercurial-scm.org/D7033