py3: make stdout line-buffered if connected to a TTY
Status messages that are to be shown on the terminal should be written to the
file descriptor before anything further is done, to keep the user updated.
One common way to achieve this is to make stdout line-buffered if it is
connected to a TTY. This is done on Python 2 (except on Windows, where libc,
which the CPython 2 streams depend on, does not properly support this).
Python 3 rolls it own I/O streams. On Python 3, buffered binary streams can't be
set line-buffered. The previous code (added in
227ba1afcb65) incorrectly
assumed that on Python 3, pycompat.stdout (sys.stdout.buffer) is already
line-buffered. However the interpreter initializes it with a block-buffered
stream or an unbuffered stream (when the -u option or the PYTHONUNBUFFERED
environment variable is set), never with a line-buffered stream.
One example where the current behavior is unacceptable is when running
`hg pull https://www.mercurial-scm.org/repo/hg` on Python 3, where the line
"pulling from https://www.mercurial-scm.org/repo/hg" does not appear on the
terminal before the hg process blocks while waiting for the server.
Various approaches to fix this problem are possible, including:
1. Weaken the contract of procutil.stdout to not give any guarantees about
buffering behavior. In this case, users of procutil.stdout need to be
changed to do enough flushes. In particular,
1. either ui must insert enough flushes for ui.write() and friends, or
2. ui.write() and friends get split into flushing and fully buffered
methods, or
3. users of ui.write() and friends must flush explicitly.
2. Make stdout unbuffered.
3. Make stdout line-buffered. Since Python 3 does not natively support that for
binary streams, we must implement it ourselves.
(2.) is problematic because using unbuffered I/O changes the performance
characteristics significantly compared to line-buffered (which is used on
Python 2) and this would be a regression.
(1.2.) and (1.3) are a substantial amount of work. It’s unclear whether the
added complexity would be justified, given that raw performance doesn’t matter
that much when writing to a terminal much faster than the user could read it.
(1.1.) pushes complexity into the ui class instead of separating the concern of
how stdout is buffered. Other users of procutil.stdout would still need to take
care of the flushes.
This patch implements (3.). The general performance considerations are very
similar to (1.1.). The extra method invocation and method forwarding add a
little more overhead if the class is used. In exchange, it doesn’t add overhead
if not used.
For the benchmarks, I compared the previous implementation (incorrect on Python
3), (1.1.), (3.) and (2.). The command was chosen so that the streams were
configured as if they were writing to a TTY, but actually write to a pager,
which is also the default:
HGRCPATH=/dev/null python3 ./hg --cwd ~/vcs/mozilla-central --time --pager yes --config pager.pager='cat > /dev/null' status --all
previous:
time: real 7.880 secs (user 7.290+0.050 sys 0.580+0.170)
time: real 7.830 secs (user 7.220+0.070 sys 0.590+0.140)
time: real 7.800 secs (user 7.210+0.050 sys 0.570+0.170)
(1.1.) using Yuya Nishihara’s patch:
time: real 9.860 secs (user 8.670+0.350 sys 1.160+0.830)
time: real 9.540 secs (user 8.430+0.370 sys 1.100+0.770)
time: real 9.830 secs (user 8.630+0.370 sys 1.180+0.840)
(3.) using this patch:
time: real 9.580 secs (user 8.480+0.350 sys 1.090+0.770)
time: real 9.670 secs (user 8.480+0.330 sys 1.170+0.860)
time: real 9.640 secs (user 8.500+0.350 sys 1.130+0.810)
(2.) using a previous patch by me:
time: real 10.480 secs (user 8.850+0.720 sys 1.590+1.500)
time: real 10.490 secs (user 8.750+0.750 sys 1.710+1.470)
time: real 10.240 secs (user 8.600+0.700 sys 1.590+1.510)
As expected, there’s no difference on Python 2, as exactly the same code paths
are used:
previous:
time: real 6.950 secs (user 5.870+0.330 sys 1.070+0.770)
time: real 7.040 secs (user 6.040+0.360 sys 0.980+0.750)
time: real 7.070 secs (user 5.950+0.360 sys 1.100+0.760)
this patch:
time: real 7.010 secs (user 5.900+0.390 sys 1.070+0.730)
time: real 7.000 secs (user 5.850+0.350 sys 1.120+0.760)
time: real 7.000 secs (user 5.790+0.380 sys 1.170+0.710)
$ cat <<EOF > merge
> from __future__ import print_function
> import sys, os
>
> try:
> import msvcrt
> msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
> msvcrt.setmode(sys.stderr.fileno(), os.O_BINARY)
> except ImportError:
> pass
>
> print("merging for", os.path.basename(sys.argv[1]))
> EOF
$ HGMERGE="\"$PYTHON\" ../merge"; export HGMERGE
$ hg init t
$ cd t
$ echo This is file a1 > a
$ hg add a
$ hg commit -m "commit #0"
$ echo This is file b1 > b
$ hg add b
$ hg commit -m "commit #1"
$ hg update 0
0 files updated, 0 files merged, 1 files removed, 0 files unresolved
Test interrupted updates by having a non-empty dir with the same name as one
of the files in a commit we're updating to
$ mkdir b && touch b/nonempty
$ hg up
abort: Unlinking directory not permitted: *$TESTTMP/t/b* (glob) (windows !)
abort: Directory not empty: '?\$TESTTMP/t/b'? (re) (no-windows !)
[255]
$ hg ci
abort: last update was interrupted
(use 'hg update' to get a consistent checkout)
[255]
$ hg sum
parent: 0:538afb845929
commit #0
branch: default
commit: 1 unknown (interrupted update)
update: 1 new changesets (update)
phases: 2 draft
Detect interrupted update by hg status --verbose
$ hg status -v
? b/nonempty
# The repository is in an unfinished *update* state.
# To continue: hg update .
$ rm b/nonempty
$ hg up
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg sum
parent: 1:b8bb4a988f25 tip
commit #1
branch: default
commit: (clean)
update: (current)
phases: 2 draft
Prepare a basic merge
$ hg up 0
0 files updated, 0 files merged, 1 files removed, 0 files unresolved
$ echo This is file c1 > c
$ hg add c
$ hg commit -m "commit #2"
created new head
$ echo This is file b1 > b
no merges expected
$ hg merge -P 1
changeset: 1:b8bb4a988f25
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: commit #1
$ hg merge 1
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ hg diff --nodates
diff -r 49035e18a8e6 b
--- /dev/null
+++ b/b
@@ -0,0 +1,1 @@
+This is file b1
$ hg status
M b
$ cd ..; rm -r t
$ hg init t
$ cd t
$ echo This is file a1 > a
$ hg add a
$ hg commit -m "commit #0"
$ echo This is file b1 > b
$ hg add b
$ hg commit -m "commit #1"
$ hg update 0
0 files updated, 0 files merged, 1 files removed, 0 files unresolved
$ echo This is file c1 > c
$ hg add c
$ hg commit -m "commit #2"
created new head
$ echo This is file b2 > b
merge should fail
$ hg merge 1
b: untracked file differs
abort: untracked files in working directory differ from files in requested revision
[255]
#if symlink
symlinks to directories should be treated as regular files (issue5027)
$ rm b
$ ln -s 'This is file b2' b
$ hg merge 1
b: untracked file differs
abort: untracked files in working directory differ from files in requested revision
[255]
symlinks shouldn't be followed
$ rm b
$ echo This is file b1 > .hg/b
$ ln -s .hg/b b
$ hg merge 1
b: untracked file differs
abort: untracked files in working directory differ from files in requested revision
[255]
$ rm b
$ echo This is file b2 > b
#endif
bad config
$ hg merge 1 --config merge.checkunknown=x
abort: merge.checkunknown not valid ('x' is none of 'abort', 'ignore', 'warn')
[255]
this merge should fail
$ hg merge 1 --config merge.checkunknown=abort
b: untracked file differs
abort: untracked files in working directory differ from files in requested revision
[255]
this merge should warn
$ hg merge 1 --config merge.checkunknown=warn
b: replacing untracked file
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ cat b.orig
This is file b2
$ hg up --clean 2
0 files updated, 0 files merged, 1 files removed, 0 files unresolved
$ mv b.orig b
this merge should silently ignore
$ cat b
This is file b2
$ hg merge 1 --config merge.checkunknown=ignore
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
merge.checkignored
$ hg up --clean 1
1 files updated, 0 files merged, 1 files removed, 0 files unresolved
$ cat >> .hgignore << EOF
> remoteignored
> EOF
$ echo This is file localignored3 > localignored
$ echo This is file remoteignored3 > remoteignored
$ hg add .hgignore localignored remoteignored
$ hg commit -m "commit #3"
$ hg up 2
1 files updated, 0 files merged, 4 files removed, 0 files unresolved
$ cat >> .hgignore << EOF
> localignored
> EOF
$ hg add .hgignore
$ hg commit -m "commit #4"
remote .hgignore shouldn't be used for determining whether a file is ignored
$ echo This is file remoteignored4 > remoteignored
$ hg merge 3 --config merge.checkignored=ignore --config merge.checkunknown=abort
remoteignored: untracked file differs
abort: untracked files in working directory differ from files in requested revision
[255]
$ hg merge 3 --config merge.checkignored=abort --config merge.checkunknown=ignore
merging .hgignore
merging for .hgignore
3 files updated, 1 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ cat remoteignored
This is file remoteignored3
$ cat remoteignored.orig
This is file remoteignored4
$ rm remoteignored.orig
local .hgignore should be used for that
$ hg up --clean 4
1 files updated, 0 files merged, 3 files removed, 0 files unresolved
$ echo This is file localignored4 > localignored
also test other conflicting files to see we output the full set of warnings
$ echo This is file b2 > b
$ hg merge 3 --config merge.checkignored=abort --config merge.checkunknown=abort
b: untracked file differs
localignored: untracked file differs
abort: untracked files in working directory differ from files in requested revision
[255]
$ hg merge 3 --config merge.checkignored=abort --config merge.checkunknown=ignore
localignored: untracked file differs
abort: untracked files in working directory differ from files in requested revision
[255]
$ hg merge 3 --config merge.checkignored=warn --config merge.checkunknown=abort
b: untracked file differs
abort: untracked files in working directory differ from files in requested revision
[255]
$ hg merge 3 --config merge.checkignored=warn --config merge.checkunknown=warn
b: replacing untracked file
localignored: replacing untracked file
merging .hgignore
merging for .hgignore
3 files updated, 1 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ cat localignored
This is file localignored3
$ cat localignored.orig
This is file localignored4
$ rm localignored.orig
$ cat b.orig
This is file b2
$ hg up --clean 2
0 files updated, 0 files merged, 4 files removed, 0 files unresolved
$ mv b.orig b
this merge of b should work
$ cat b
This is file b2
$ hg merge -f 1
merging b
merging for b
0 files updated, 1 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ hg diff --nodates
diff -r 49035e18a8e6 b
--- /dev/null
+++ b/b
@@ -0,0 +1,1 @@
+This is file b2
$ hg status
M b
$ cd ..; rm -r t
$ hg init t
$ cd t
$ echo This is file a1 > a
$ hg add a
$ hg commit -m "commit #0"
$ echo This is file b1 > b
$ hg add b
$ hg commit -m "commit #1"
$ echo This is file b22 > b
$ hg commit -m "commit #2"
$ hg update 1
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ echo This is file c1 > c
$ hg add c
$ hg commit -m "commit #3"
created new head
Contents of b should be "this is file b1"
$ cat b
This is file b1
$ echo This is file b22 > b
merge fails
$ hg merge 2
abort: uncommitted changes
(use 'hg status' to list changes)
[255]
merge expected!
$ hg merge -f 2
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ hg diff --nodates
diff -r 85de557015a8 b
--- a/b
+++ b/b
@@ -1,1 +1,1 @@
-This is file b1
+This is file b22
$ hg status
M b
$ cd ..; rm -r t
$ hg init t
$ cd t
$ echo This is file a1 > a
$ hg add a
$ hg commit -m "commit #0"
$ echo This is file b1 > b
$ hg add b
$ hg commit -m "commit #1"
$ echo This is file b22 > b
$ hg commit -m "commit #2"
$ hg update 1
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ echo This is file c1 > c
$ hg add c
$ hg commit -m "commit #3"
created new head
$ echo This is file b33 > b
merge of b should fail
$ hg merge 2
abort: uncommitted changes
(use 'hg status' to list changes)
[255]
merge of b expected
$ hg merge -f 2
merging b
merging for b
0 files updated, 1 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ hg diff --nodates
diff -r 85de557015a8 b
--- a/b
+++ b/b
@@ -1,1 +1,1 @@
-This is file b1
+This is file b33
$ hg status
M b
Test for issue2364
$ hg up -qC .
$ hg rm b
$ hg ci -md
$ hg revert -r -2 b
$ hg up -q -- -2
Test that updated files are treated as "modified", when
'merge.update()' is aborted before 'merge.recordupdates()' (= parents
aren't changed), even if none of mode, size and timestamp of them
isn't changed on the filesystem (see also issue4583).
$ cat > $TESTTMP/abort.py <<EOF
> from __future__ import absolute_import
> # emulate aborting before "recordupdates()". in this case, files
> # are changed without updating dirstate
> from mercurial import (
> error,
> extensions,
> merge,
> )
> def applyupdates(orig, *args, **kwargs):
> orig(*args, **kwargs)
> raise error.Abort('intentional aborting')
> def extsetup(ui):
> extensions.wrapfunction(merge, "applyupdates", applyupdates)
> EOF
$ cat >> .hg/hgrc <<EOF
> [fakedirstatewritetime]
> # emulate invoking dirstate.write() via repo.status()
> # at 2000-01-01 00:00
> fakenow = 200001010000
> EOF
(file gotten from other revision)
$ hg update -q -C 2
$ echo 'THIS IS FILE B5' > b
$ hg commit -m 'commit #5'
$ hg update -q -C 3
$ cat b
This is file b1
$ touch -t 200001010000 b
$ hg debugrebuildstate
$ cat >> .hg/hgrc <<EOF
> [extensions]
> fakedirstatewritetime = $TESTDIR/fakedirstatewritetime.py
> abort = $TESTTMP/abort.py
> EOF
$ hg merge 5
abort: intentional aborting
[255]
$ cat >> .hg/hgrc <<EOF
> [extensions]
> fakedirstatewritetime = !
> abort = !
> EOF
$ cat b
THIS IS FILE B5
$ touch -t 200001010000 b
$ hg status -A b
M b
(file merged from other revision)
$ hg update -q -C 3
$ echo 'this is file b6' > b
$ hg commit -m 'commit #6'
created new head
$ cat b
this is file b6
$ touch -t 200001010000 b
$ hg debugrebuildstate
$ cat >> .hg/hgrc <<EOF
> [extensions]
> fakedirstatewritetime = $TESTDIR/fakedirstatewritetime.py
> abort = $TESTTMP/abort.py
> EOF
$ hg merge --tool internal:other 5
abort: intentional aborting
[255]
$ cat >> .hg/hgrc <<EOF
> [extensions]
> fakedirstatewritetime = !
> abort = !
> EOF
$ cat b
THIS IS FILE B5
$ touch -t 200001010000 b
$ hg status -A b
M b
$ cd ..