Manuel Jacob <me@manueljacob.de> [Wed, 10 Jun 2020 13:02:39 +0200] rev 44991
py3: make stdout line-buffered if connected to a TTY
Status messages that are to be shown on the terminal should be written to the
file descriptor before anything further is done, to keep the user updated.
One common way to achieve this is to make stdout line-buffered if it is
connected to a TTY. This is done on Python 2 (except on Windows, where libc,
which the CPython 2 streams depend on, does not properly support this).
Python 3 rolls it own I/O streams. On Python 3, buffered binary streams can't be
set line-buffered. The previous code (added in 227ba1afcb65) incorrectly
assumed that on Python 3, pycompat.stdout (sys.stdout.buffer) is already
line-buffered. However the interpreter initializes it with a block-buffered
stream or an unbuffered stream (when the -u option or the PYTHONUNBUFFERED
environment variable is set), never with a line-buffered stream.
One example where the current behavior is unacceptable is when running
`hg pull https://www.mercurial-scm.org/repo/hg` on Python 3, where the line
"pulling from https://www.mercurial-scm.org/repo/hg" does not appear on the
terminal before the hg process blocks while waiting for the server.
Various approaches to fix this problem are possible, including:
1. Weaken the contract of procutil.stdout to not give any guarantees about
buffering behavior. In this case, users of procutil.stdout need to be
changed to do enough flushes. In particular,
1. either ui must insert enough flushes for ui.write() and friends, or
2. ui.write() and friends get split into flushing and fully buffered
methods, or
3. users of ui.write() and friends must flush explicitly.
2. Make stdout unbuffered.
3. Make stdout line-buffered. Since Python 3 does not natively support that for
binary streams, we must implement it ourselves.
(2.) is problematic because using unbuffered I/O changes the performance
characteristics significantly compared to line-buffered (which is used on
Python 2) and this would be a regression.
(1.2.) and (1.3) are a substantial amount of work. It’s unclear whether the
added complexity would be justified, given that raw performance doesn’t matter
that much when writing to a terminal much faster than the user could read it.
(1.1.) pushes complexity into the ui class instead of separating the concern of
how stdout is buffered. Other users of procutil.stdout would still need to take
care of the flushes.
This patch implements (3.). The general performance considerations are very
similar to (1.1.). The extra method invocation and method forwarding add a
little more overhead if the class is used. In exchange, it doesn’t add overhead
if not used.
For the benchmarks, I compared the previous implementation (incorrect on Python
3), (1.1.), (3.) and (2.). The command was chosen so that the streams were
configured as if they were writing to a TTY, but actually write to a pager,
which is also the default:
HGRCPATH=/dev/null python3 ./hg --cwd ~/vcs/mozilla-central --time --pager yes --config pager.pager='cat > /dev/null' status --all
previous:
time: real 7.880 secs (user 7.290+0.050 sys 0.580+0.170)
time: real 7.830 secs (user 7.220+0.070 sys 0.590+0.140)
time: real 7.800 secs (user 7.210+0.050 sys 0.570+0.170)
(1.1.) using Yuya Nishihara’s patch:
time: real 9.860 secs (user 8.670+0.350 sys 1.160+0.830)
time: real 9.540 secs (user 8.430+0.370 sys 1.100+0.770)
time: real 9.830 secs (user 8.630+0.370 sys 1.180+0.840)
(3.) using this patch:
time: real 9.580 secs (user 8.480+0.350 sys 1.090+0.770)
time: real 9.670 secs (user 8.480+0.330 sys 1.170+0.860)
time: real 9.640 secs (user 8.500+0.350 sys 1.130+0.810)
(2.) using a previous patch by me:
time: real 10.480 secs (user 8.850+0.720 sys 1.590+1.500)
time: real 10.490 secs (user 8.750+0.750 sys 1.710+1.470)
time: real 10.240 secs (user 8.600+0.700 sys 1.590+1.510)
As expected, there’s no difference on Python 2, as exactly the same code paths
are used:
previous:
time: real 6.950 secs (user 5.870+0.330 sys 1.070+0.770)
time: real 7.040 secs (user 6.040+0.360 sys 0.980+0.750)
time: real 7.070 secs (user 5.950+0.360 sys 1.100+0.760)
this patch:
time: real 7.010 secs (user 5.900+0.390 sys 1.070+0.730)
time: real 7.000 secs (user 5.850+0.350 sys 1.120+0.760)
time: real 7.000 secs (user 5.790+0.380 sys 1.170+0.710)
Yuya Nishihara <yuya@tcha.org> [Tue, 02 Jun 2020 21:44:57 +0900] rev 44990
simplemerge: rewrite flag merging loop as expression
I feel binary operations are more readable.
Yuya Nishihara <yuya@tcha.org> [Tue, 02 Jun 2020 21:40:49 +0900] rev 44989
simplemerge: leverage pycompat function to convert byte string to set
Yuya Nishihara <yuya@tcha.org> [Tue, 02 Jun 2020 21:39:07 +0900] rev 44988
simplemerge: fix function name that tests if ctx is not null revision
Hollis Blanchard <hollis_blanchard@mentor.com> [Tue, 09 Jun 2020 13:18:21 -0700] rev 44987
git: decode node IDs back into Python strings (issue6349)
db.text_factory = bytes, so the database contains only strings. The object IDs
we get from pygit2 are Python strings. b'foo' != 'foo'
This change allows the "don't reindex" optimization to work by allowing the
"cur_cache_heads == cache_heads" comparison a few lines down to succeed.
Differential Revision: https://phab.mercurial-scm.org/D8622
Sushil khanchi <sushilkhanchi97@gmail.com> [Tue, 09 Jun 2020 22:02:09 +0530] rev 44986
phabricator: make it clear what happen when no response
Differential Revision: https://phab.mercurial-scm.org/D8621
Sushil khanchi <sushilkhanchi97@gmail.com> [Mon, 08 Jun 2020 11:43:07 +0530] rev 44985
tests: make it clear what happen when no response entered
Differential Revision: https://phab.mercurial-scm.org/D8620
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 18 Jan 2020 10:07:07 -0800] rev 44984
localrepo: handle ValueError during repository opening
Python 3.8 can raise ValueError on attempt of an I/O operation
against an illegal path. This was causing test-remotefilelog-gc.t
to fail on Python 3.8.
This commit teaches repository opening to handle ValueError
and re-raise an Abort on failure.
An arguably better solution would be to implement this logic
in the vfs layer. But that seems like a bag of worms and I don't
want to go down that rabbit hole. Until users report uncaught
ValueError exceptions in the wild, I think it is fine to patch
this at the only occurrence our test harness is finding it.
Differential Revision: https://phab.mercurial-scm.org/D7944
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 27 May 2020 12:56:13 +0200] rev 44983
metadata: filter the `removed` set to only contains relevant data
The `files` entry can be bogus and contains too many entries. This can badly
combines with the computation of `removed` inflating the set size. The can lead
to the changesets centric rename computation to process much more data than
needed, slowing it down (and increasing space taken by data storage).
In practice newer commits already that reduced set, this applies this "fix" to
older changeset.
Differential Revision: https://phab.mercurial-scm.org/D8589
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 27 May 2020 12:45:39 +0200] rev 44982
files: extract code for extra filtering of the `removed` entry into copies
We want to reduce the set of `removed` files that to the set of files actually
removed. That `removed` set is used as of the changeset centric algorithm,
having smaller sets means less processing and faster computation.
In this changeset we extract the code to be a function of it own. We will make
use of it in the next changesets.
Differential Revision: https://phab.mercurial-scm.org/D8588
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 27 May 2020 12:26:08 +0200] rev 44981
metadata: move computation related to files touched in a dedicated module
This was suggested by Yuya Nishihara a while back. Since I am about to add more
metadata related computation, lets create a new repositories.
Differential Revision: https://phab.mercurial-scm.org/D8587
Martin von Zweigbergk <martinvonz@google.com> [Fri, 05 Jun 2020 11:10:33 -0700] rev 44980
merge: move an inspection of the dirstate from record to calculate phase
The intent is clearly to have `calculateupdates()` figure out what
actions need to be taken and `recordupdates()` to make necessary
modifications to the dirstate. However, in the
`ACTION_PATH_CONFLICT_RESOLVE` case, there was one little inspection
of copy information done in `recordupdates()`. This patch moves that
to `calculateupdates()`. That will help with the next patch, which
makes `merge.update()` work better with `overlayworkingctx` (copies
should be recorded there too, even though we skip the
`recordupdates()` step).
Differential Revision: https://phab.mercurial-scm.org/D8615
Anton Shestakov <av6@dwimlabs.net> [Sat, 06 Jun 2020 19:15:11 +0800] rev 44979
tests: adjust to the new format in pyflakes output
According to the pyflakes' NEWS.rst, the default output format changed
recently:
2.2.0 (2020-04-08)
- Include column information in error messages
So the lines now read:
contrib/perf.py:149:15 undefined name 'xrange'
mercurial/hgweb/server.py:427:13 undefined name 'reload'
mercurial/util.py:2862:24 undefined name 'file'
Differential Revision: https://phab.mercurial-scm.org/D8618
Anton Shestakov <av6@dwimlabs.net> [Sat, 06 Jun 2020 19:12:49 +0800] rev 44978
tests: consistently use pyflakes as a Python module
We check availability of pyflakes as a module, and also running it for real as
a module. Only fair to test filterpyflakes.py working correctly when using
pyflakes as a module too.
Differential Revision: https://phab.mercurial-scm.org/D8617
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 26 Feb 2020 00:31:23 +0100] rev 44977
heptapod-ci: also run tests for chg on python 2
I am not aware of any regular effort to run test with chg. So let's at least do it
here.
Differential Revision: https://phab.mercurial-scm.org/D8178
Raphaël Gomès <rgomes@octobus.net> [Tue, 02 Jun 2020 17:24:37 +0200] rev 44976
rust-dependencies: upgrade `micro-timer` dependency
I wanted to to a tour of dependencies to upgrade, but only `micro-timer` has a
new release which does not print when the function panics, which should be less
misleading.
Differential Revision: https://phab.mercurial-scm.org/D8605
Martin von Zweigbergk <martinvonz@google.com> [Wed, 03 Jun 2020 12:04:38 -0700] rev 44975
context: fix creation of ProgrammingError to not use non-existent field
Differential Revision: https://phab.mercurial-scm.org/D8606
Martin von Zweigbergk <martinvonz@google.com> [Wed, 03 Jun 2020 22:07:26 -0700] rev 44974
help: explain in `hg help flags` that unambiguous prefixes are allowed
I used `hg commit --amend` as an example because that's the most
frequently used flag I could think of that doesn't yet have a short
form.
Differential Revision: https://phab.mercurial-scm.org/D8607
Augie Fackler <augie@google.com> [Wed, 03 Jun 2020 11:28:57 -0400] rev 44973
git: add coverage for manifest.diff() so we don't regress
Josef 'Jeff' Sipek <jeffpc@josefsipek.net> [Mon, 01 Jun 2020 11:12:25 -0400] rev 44972
git: implement diff manifest method
This makes 'hg diff' work.
Josef 'Jeff' Sipek <jeffpc@josefsipek.net> [Mon, 01 Jun 2020 09:49:47 -0400] rev 44971
git: properly visit child tree objects when resolving a path
Josef 'Jeff' Sipek <jeffpc@josefsipek.net> [Mon, 01 Jun 2020 09:40:18 -0400] rev 44970
git: don't yield paths for directories when walking
Josef 'Jeff' Sipek <jeffpc@josefsipek.net> [Mon, 01 Jun 2020 09:22:53 -0400] rev 44969
git: correctly check for type of object when walking
Josef 'Jeff' Sipek <jeffpc@josefsipek.net> [Mon, 01 Jun 2020 08:59:48 -0400] rev 44968
git: implement stub prefetch_parents dirstate method
A recent change (35b255e474d9) introduced this new required dirstate method
but didn't update the git extension.
Yuya Nishihara <yuya@tcha.org> [Mon, 25 May 2020 23:06:50 +0900] rev 44967
rust: leverage .expect() in place of .unwrap() + inline comment
For a better error indication in case we made a mistake.
Yuya Nishihara <yuya@tcha.org> [Mon, 25 May 2020 23:02:07 +0900] rev 44966
rust: fix false comment about mpsc::Sender
We need Sync to share the Sender reference across threads.
Yuya Nishihara <yuya@tcha.org> [Wed, 03 Jun 2020 19:20:18 +0900] rev 44965
merge with stable
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Sat, 30 May 2020 12:36:00 -0400] rev 44964
relnotes: advertize the possibility to use rust
I think the rust work may have been mentioned in the release notes,
but if so only in passing, and not as an invitation to try it out.
I think the next version is a decent time to do this, because the rust
doesn't come with performance regressions AFAIK, speeds up status
noticeably when it applies, which is the case for most invocations of
status, and doesn't have the undesirable restriction of regex around
empty patterns anymore.
I am cheating a bit, because I'm giving numbers for `hg status` in
mozilla-central, but they have one hgignore pattern that uses
lookaround, ".vscode/(?!extensions\.json|tasks\.json", which I took
out as it would cause a fallback to python when unknown files are
requested. But it seems that they could express their hgignore
differently if they were so inclined.
Not sure if there are limitation other than linux-only that I am
not thinking of but would be worth mentioning upfront, to avoid
disappointing users?
Differential Revision: https://phab.mercurial-scm.org/D8604
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Sat, 30 May 2020 11:36:30 -0400] rev 44963
rust: add a pointer for profiling to the README
As figuring out how to get useful profiles is not obvious.
Differential Revision: https://phab.mercurial-scm.org/D8603
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Sat, 30 May 2020 10:28:46 -0400] rev 44962
rust: update the mention of hgcli in rust/README.rst
This may not be exactly right, but it's better than before.
Differential Revision: https://phab.mercurial-scm.org/D8602
Manuel Jacob <me@manueljacob.de> [Mon, 01 Jun 2020 15:22:31 +0200] rev 44961
sslutil: fix comment to use inclusive or instead of exclusive or
The incorrect "either" was introduced by one of my recent patches.
Manuel Jacob <me@manueljacob.de> [Mon, 01 Jun 2020 14:34:22 +0200] rev 44960
sslutil: propagate return value ssl.PROTOCOL_SSLv23 from protocolsettings()
Also, protocolsettings() was renamed to commonssloptions() to reflect that
only the options are returned.
Manuel Jacob <me@manueljacob.de> [Mon, 01 Jun 2020 14:20:13 +0200] rev 44959
sslutil: stop storing protocol and options for SSLContext in settings dict
Call protocolsettings() where its return values are needed.
Manuel Jacob <me@manueljacob.de> [Mon, 01 Jun 2020 14:07:06 +0200] rev 44958
sslutil: rename 'minimumprotocolui' -> 'minimumprotocol'
Before, both 'minimumprotocolui' and 'minimumprotocol' were used, but meaning
the same.
Manuel Jacob <me@manueljacob.de> [Mon, 01 Jun 2020 03:51:54 +0200] rev 44957
sslutil: properly detect which TLS versions are supported by the ssl module
For the record, I contacted the CPython developers to remark that
unconditionally defining ssl.PROTOCOL_TLSv1_1 / ssl.PROTOCOL_TLSv1_2 is
problematic:
https://github.com/python/cpython/commit/6e8cda91d92da72800d891b2fc2073ecbc134d98#r39569316
Manuel Jacob <me@manueljacob.de> [Sun, 31 May 2020 22:31:49 +0200] rev 44956
sslutil: remove dead code (that failed if only TLS 1.0 is available)
We ensure in setup.py that TLS 1.1 or TLS 1.2 is present.
Manuel Jacob <me@manueljacob.de> [Sun, 31 May 2020 00:30:49 +0200] rev 44955
config: remove unused hostsecurity.disabletls10warning config
Manuel Jacob <me@manueljacob.de> [Sun, 31 May 2020 22:15:35 +0200] rev 44954
sslutil: remove dead code (that downgraded default minimum TLS version)
We ensure in setup.py that TLS 1.1 or TLS 1.2 is present.
Manuel Jacob <me@manueljacob.de> [Fri, 29 May 2020 22:47:58 +0200] rev 44953
sslutil: remove comment referring to unsupported legacy stacks
Manuel Jacob <me@manueljacob.de> [Sat, 30 May 2020 23:42:19 +0200] rev 44952
setup: require that Python has TLS 1.1 or TLS 1.2
This ensures that Mercurial never downgrades the minimum TLS version from
TLS 1.1+ to TLS 1.0+ and enables us to remove that compatibility code.
It is reasonable to expect that distributions having Python 2.7.9+ or having
backported modern features to the ssl module (which we require) have a OpenSSL
version supporting TLS 1.1 or TLS 1.2, as this is the main reason why
distributions would want to backport these features.
TLS 1.1 and TLS 1.2 are often either both enabled or both not enabled.
However, both can be disabled independently, at least on current Python /
OpenSSL versions.
For the record, I contacted the CPython developers to remark that
unconditionally defining ssl.PROTOCOL_TLSv1_1 / ssl.PROTOCOL_TLSv1_2 is
problematic:
https://github.com/python/cpython/commit/6e8cda91d92da72800d891b2fc2073ecbc134d98#r39569316
Manuel Jacob <me@manueljacob.de> [Sun, 31 May 2020 12:07:17 +0200] rev 44951
sslutil: check for OpenSSL without TLS 1.0 support in one case
It can only happen if supportedprotocols gets fixed to contain only correct
items (see the FIXME above in the file).
Manuel Jacob <me@manueljacob.de> [Sun, 31 May 2020 11:10:21 +0200] rev 44950
sslutil: don't set minimum TLS version to 1.0 if 1.2 but not 1.1 is available
This case isn't very likely, but possible, especially if supportedprotocols
gets fixed to contain only correct items (see the FIXME above in the file).
Manuel Jacob <me@manueljacob.de> [Sun, 31 May 2020 11:41:03 +0200] rev 44949
sslutil: add FIXME about supportedprotocols possibly containing too many items
Manuel Jacob <me@manueljacob.de> [Sun, 31 May 2020 10:47:38 +0200] rev 44948
sslutil: fix names of variables containing minimum protocol strings
When working in this module, I found it very confusing that "protocol" as a
variable name could mean either "minimum protocol string" or an exact version
(as a string or ssl.PROTOCOL_* value). This patch prefixes variables of the
former type with "minimum".
Manuel Jacob <me@manueljacob.de> [Sun, 31 May 2020 09:55:45 +0200] rev 44947
sslutil: stop returning argument as third return value of protocolsettings()
The third return value was always the same as the argument.
Manuel Jacob <me@manueljacob.de> [Sat, 30 May 2020 23:18:57 +0200] rev 44946
relnotes: note that we now require modern SSL/TLS features in Python
Manuel Jacob <me@manueljacob.de> [Sat, 30 May 2020 19:04:53 +0200] rev 44945
tests: stop checking for optional, now impossible output
In 7dd63a8cb1ee, the code that could output that line was removed.
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Sat, 30 May 2020 10:19:53 -0400] rev 44944
rust: remove one more occurrence of re2
Differential Revision: https://phab.mercurial-scm.org/D8601
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Tue, 26 May 2020 07:03:11 -0400] rev 44943
scmutil: clarify getuipathfn comment
Differential Revision: https://phab.mercurial-scm.org/D8600
Augie Fackler <augie@google.com> [Thu, 28 May 2020 09:51:13 -0400] rev 44942
githelp: add some minimal help for pickaxe functionality
Due to a conversation in work chat, I realized this is actually pretty
well-hidden in Mercurial.
Differential Revision: https://phab.mercurial-scm.org/D8590
Raphaël Gomès <rgomes@octobus.net> [Fri, 17 Apr 2020 10:41:05 +0200] rev 44941
rust: remove duplicate import
Differential Revision: https://phab.mercurial-scm.org/D8456
Manuel Jacob <me@manueljacob.de> [Sat, 30 May 2020 05:27:53 +0200] rev 44940
tests: remove "sslcontext" check
Now that we require the presence of ssl.SSLContext in setup.py, the check
would always return `True`.
Manuel Jacob <me@manueljacob.de> [Sat, 30 May 2020 03:23:58 +0200] rev 44939
sslutil: eliminate `_canloaddefaultcerts` by constant-folding code using it
Manuel Jacob <me@manueljacob.de> [Sat, 30 May 2020 05:08:02 +0200] rev 44938
tests: remove "defaultcacerts" check
`sslutil._canloaddefaultcerts` is always true (and will be removed).
Manuel Jacob <me@manueljacob.de> [Fri, 29 May 2020 21:30:04 +0200] rev 44937
sslutil: eliminate `modernssl` by constant-folding code using it
Manuel Jacob <me@manueljacob.de> [Sat, 30 May 2020 04:59:13 +0200] rev 44936
hgweb: avoid using `sslutil.modernssl`
`sslutil.modernssl` is going to be removed. Since the point of using this
attribute was to check the importability of the `sslutil`, a different
attribute can be used. `sslutil.wrapserversocket` is used because it’s anyway
used a few lines below.
Manuel Jacob <me@manueljacob.de> [Fri, 29 May 2020 22:31:26 +0200] rev 44935
sslutil: remove comments referring to removed SSLContext emulation class
Manuel Jacob <me@manueljacob.de> [Fri, 29 May 2020 21:18:22 +0200] rev 44934
sslutil: remove code checking for presence of ssl.SSLContext
Now that we require the presence of ssl.SSLContext in setup.py, we can remove
this code.
Manuel Jacob <me@manueljacob.de> [Fri, 29 May 2020 21:07:26 +0200] rev 44933
setup: require a Python version with modern SSL features
This increases the minimum security baseline of Mercurial and enables us to
remove compatibility code for supporting older, less secure Python versions.
Manuel Jacob <me@manueljacob.de> [Sat, 30 May 2020 03:46:59 +0200] rev 44932
sslutil: set `_canloaddefaultcerts` to `True` if `ssl.SSLContext` is present
The `load_default_certs()` method was already present when `ssl.SSLContext`
was backported to Python 2.7 (https://hg.python.org/cpython/rev/221a1f9155e2).
Augie Fackler <augie@google.com> [Thu, 28 May 2020 16:16:13 -0400] rev 44931
filemerge: add __bytes__ for absentfilectx
This will at _least_ aid some upcoming debugging.
Differential Revision: https://phab.mercurial-scm.org/D8592
Augie Fackler <augie@google.com> [Thu, 28 May 2020 16:17:28 -0400] rev 44930
mergestate: move staticmethod _filectxorabsent to module level
I suspect this was a static method just because it made merge.py feel
less messy, but now we have a mergestate package so we can do better.
Differential Revision: https://phab.mercurial-scm.org/D8591
Raphaël Gomès <rgomes@octobus.net> [Fri, 29 May 2020 12:17:59 +0200] rev 44929
rust: remove support for `re2`
With the performance issues with `regex` figured out and fixed in previous
patches and `regex` newly gaining support for empty alternations, there is no
reason to keep `re2` around anymore. It's only *marginally* faster at creating
the regex which saves at most a couple of ms, but gets beaten by `regex` in
every other aspect.
This removes the Rust/C/C++ bridge (hooray!), the `with-re2` feature, the
conditional code that goes with it, the documentation and relevant part of the
debug/module output.
Differential Revision: https://phab.mercurial-scm.org/D8594
Raphaël Gomès <rgomes@octobus.net> [Fri, 29 May 2020 12:12:16 +0200] rev 44928
rust-dependencies: update `regex` to 1.3.9
Version `1.3.8` introduces support for empty alternations, which makes
previously disallowed patterns usable in `regex`.
From a user's perspective, this means that glob patterns like `*.py{,c}` will
no longer generate an "invalid" regex and will use the Rust path.
`1.3.9` is a bugfix release, might as well update to the latest one.
Differential Revision: https://phab.mercurial-scm.org/D8593