py3: make stdout line-buffered if connected to a TTY
Status messages that are to be shown on the terminal should be written to the
file descriptor before anything further is done, to keep the user updated.
One common way to achieve this is to make stdout line-buffered if it is
connected to a TTY. This is done on Python 2 (except on Windows, where libc,
which the CPython 2 streams depend on, does not properly support this).
Python 3 rolls it own I/O streams. On Python 3, buffered binary streams can't be
set line-buffered. The previous code (added in
227ba1afcb65) incorrectly
assumed that on Python 3, pycompat.stdout (sys.stdout.buffer) is already
line-buffered. However the interpreter initializes it with a block-buffered
stream or an unbuffered stream (when the -u option or the PYTHONUNBUFFERED
environment variable is set), never with a line-buffered stream.
One example where the current behavior is unacceptable is when running
`hg pull https://www.mercurial-scm.org/repo/hg` on Python 3, where the line
"pulling from https://www.mercurial-scm.org/repo/hg" does not appear on the
terminal before the hg process blocks while waiting for the server.
Various approaches to fix this problem are possible, including:
1. Weaken the contract of procutil.stdout to not give any guarantees about
buffering behavior. In this case, users of procutil.stdout need to be
changed to do enough flushes. In particular,
1. either ui must insert enough flushes for ui.write() and friends, or
2. ui.write() and friends get split into flushing and fully buffered
methods, or
3. users of ui.write() and friends must flush explicitly.
2. Make stdout unbuffered.
3. Make stdout line-buffered. Since Python 3 does not natively support that for
binary streams, we must implement it ourselves.
(2.) is problematic because using unbuffered I/O changes the performance
characteristics significantly compared to line-buffered (which is used on
Python 2) and this would be a regression.
(1.2.) and (1.3) are a substantial amount of work. It’s unclear whether the
added complexity would be justified, given that raw performance doesn’t matter
that much when writing to a terminal much faster than the user could read it.
(1.1.) pushes complexity into the ui class instead of separating the concern of
how stdout is buffered. Other users of procutil.stdout would still need to take
care of the flushes.
This patch implements (3.). The general performance considerations are very
similar to (1.1.). The extra method invocation and method forwarding add a
little more overhead if the class is used. In exchange, it doesn’t add overhead
if not used.
For the benchmarks, I compared the previous implementation (incorrect on Python
3), (1.1.), (3.) and (2.). The command was chosen so that the streams were
configured as if they were writing to a TTY, but actually write to a pager,
which is also the default:
HGRCPATH=/dev/null python3 ./hg --cwd ~/vcs/mozilla-central --time --pager yes --config pager.pager='cat > /dev/null' status --all
previous:
time: real 7.880 secs (user 7.290+0.050 sys 0.580+0.170)
time: real 7.830 secs (user 7.220+0.070 sys 0.590+0.140)
time: real 7.800 secs (user 7.210+0.050 sys 0.570+0.170)
(1.1.) using Yuya Nishihara’s patch:
time: real 9.860 secs (user 8.670+0.350 sys 1.160+0.830)
time: real 9.540 secs (user 8.430+0.370 sys 1.100+0.770)
time: real 9.830 secs (user 8.630+0.370 sys 1.180+0.840)
(3.) using this patch:
time: real 9.580 secs (user 8.480+0.350 sys 1.090+0.770)
time: real 9.670 secs (user 8.480+0.330 sys 1.170+0.860)
time: real 9.640 secs (user 8.500+0.350 sys 1.130+0.810)
(2.) using a previous patch by me:
time: real 10.480 secs (user 8.850+0.720 sys 1.590+1.500)
time: real 10.490 secs (user 8.750+0.750 sys 1.710+1.470)
time: real 10.240 secs (user 8.600+0.700 sys 1.590+1.510)
As expected, there’s no difference on Python 2, as exactly the same code paths
are used:
previous:
time: real 6.950 secs (user 5.870+0.330 sys 1.070+0.770)
time: real 7.040 secs (user 6.040+0.360 sys 0.980+0.750)
time: real 7.070 secs (user 5.950+0.360 sys 1.100+0.760)
this patch:
time: real 7.010 secs (user 5.900+0.390 sys 1.070+0.730)
time: real 7.000 secs (user 5.850+0.350 sys 1.120+0.760)
time: real 7.000 secs (user 5.790+0.380 sys 1.170+0.710)
Testing changing branch on commits
==================================
Setup
$ cat >> $HGRCPATH << EOF
> [alias]
> glog = log -G -T "{rev}:{node|short} {desc}\n{branch} ({bookmarks})"
> [experimental]
> evolution = createmarkers
> [extensions]
> rebase=
> EOF
$ hg init repo
$ cd repo
$ for ch in a b c d e; do echo foo >> $ch; hg ci -Aqm "Added "$ch; done
$ hg glog
@ 4:aa98ab95a928 Added e
| default ()
o 3:62615734edd5 Added d
| default ()
o 2:28ad74487de9 Added c
| default ()
o 1:29becc82797a Added b
| default ()
o 0:18d04c59bb5d Added a
default ()
$ hg branches
default 4:aa98ab95a928
Try without passing a new branch name
$ hg branch -r .
abort: no branch name specified for the revisions
[255]
Setting an invalid branch name
$ hg branch -r . a:b
abort: ':' cannot be used in a name
[255]
$ hg branch -r . tip
abort: the name 'tip' is reserved
[255]
$ hg branch -r . 1234
abort: cannot use an integer as a name
[255]
Change on non-linear set of commits
$ hg branch -r 2 -r 4 foo
abort: cannot change branch of non-linear revisions
[255]
Change in middle of the stack (linear commits)
$ hg branch -r 1::3 foo
abort: cannot change branch of changeset with children
[255]
Change with dirty working directory
$ echo bar > a
$ hg branch -r . foo
abort: uncommitted changes
[255]
$ hg revert --all
reverting a
Change on empty revision set
$ hg branch -r 'draft() - all()' foo
abort: empty revision set
[255]
Changing branch on linear set of commits from head
Without obsmarkers
$ hg branch -r 3:4 foo --config experimental.evolution=!
changed branch on 2 changesets
saved backup bundle to $TESTTMP/repo/.hg/strip-backup/62615734edd5-e86bd13a-branch-change.hg
$ hg glog
@ 4:3938acfb5c0f Added e
| foo ()
o 3:9435da006bdc Added d
| foo ()
o 2:28ad74487de9 Added c
| default ()
o 1:29becc82797a Added b
| default ()
o 0:18d04c59bb5d Added a
default ()
$ hg branches
foo 4:3938acfb5c0f
default 2:28ad74487de9 (inactive)
With obsmarkers
$ hg branch -r 3::4 bar
changed branch on 2 changesets
$ hg glog
@ 6:7c1991464886 Added e
| bar ()
o 5:1ea05e93925f Added d
| bar ()
o 2:28ad74487de9 Added c
| default ()
o 1:29becc82797a Added b
| default ()
o 0:18d04c59bb5d Added a
default ()
$ hg branches
bar 6:7c1991464886
default 2:28ad74487de9 (inactive)
Change branch name to an existing branch
$ hg branch -r . default
abort: a branch of the same name already exists
[255]
Changing on a branch head which is not topological head
$ hg branch -r 2 stable
abort: cannot change branch of changeset with children
[255]
Enabling the allowunstable config and trying to change branch on a branch head
which is not a topological head
$ echo "[experimental]" >> .hg/hgrc
$ echo "evolution.allowunstable=yes" >> .hg/hgrc
$ hg branch -r 2 foo
changed branch on 1 changesets
2 new orphan changesets
Changing branch of an obsoleted changeset
$ hg branch -r 4 foobar
abort: hidden revision '4' was rewritten as: 7c1991464886!
(use --hidden to access hidden revisions)
[255]
$ hg branch -r 4 --hidden foobar
abort: cannot change branch of a obsolete changeset
[255]
Make sure bookmark movement is correct
$ hg bookmark b1
$ hg glog -r '.^::'
@ 6:7c1991464886 Added e
| bar (b1)
* 5:1ea05e93925f Added d
| bar ()
~
$ hg branch -r '(.^)::' wat --debug
changing branch of '1ea05e93925f806d875a2163f9b76764be644636' from 'bar' to 'wat'
committing files:
d
committing manifest
committing changelog
new node id is 343660ccab7400da637bd6a211d07f413536d718
changing branch of '7c19914648869f5b02fc7fed31ddee9783fdd680' from 'bar' to 'wat'
committing files:
e
committing manifest
committing changelog
new node id is de1404b45a69f8cc6437d7679033ee33e9efb4ba
moving bookmarks ['b1'] from 7c19914648869f5b02fc7fed31ddee9783fdd680 to de1404b45a69f8cc6437d7679033ee33e9efb4ba
resolving manifests
branchmerge: False, force: False, partial: False
ancestor: 7c1991464886, local: 7c1991464886+, remote: de1404b45a69
starting 4 threads for background file closing (?)
changed branch on 2 changesets
updating the branch cache
invalid branch cache (served): tip differs
$ hg glog -r '(.^)::'
@ 9:de1404b45a69 Added e
| wat (b1)
* 8:343660ccab74 Added d
| wat ()
~
Make sure phase handling is correct
$ echo foo >> bar
$ hg ci -Aqm "added bar" --secret
1 new orphan changesets
$ hg glog -r .
@ 10:8ad1294c1660 added bar
| wat (b1)
~
$ hg branch -r . secret
changed branch on 1 changesets
$ hg phase -r .
11: secret
$ hg branches
secret 11:38a9b2d53f98
foo 7:8a4729a5e2b8
wat 9:de1404b45a69 (inactive)
default 2:28ad74487de9 (inactive)
$ hg branch
secret
Changing branch of another head, different from one on which we are
$ hg glog
@ 11:38a9b2d53f98 added bar
| secret (b1)
* 9:de1404b45a69 Added e
| wat ()
* 8:343660ccab74 Added d
| wat ()
| o 7:8a4729a5e2b8 Added c
| | foo ()
x | 2:28ad74487de9 Added c
|/ default ()
o 1:29becc82797a Added b
| default ()
o 0:18d04c59bb5d Added a
default ()
$ hg branch
secret
$ hg branch -r 7 foobar
changed branch on 1 changesets
The current branch must be preserved
$ hg branch
secret
Changing branch on multiple heads at once
$ hg rebase -s 8 -d 12 --keepbranches -q
$ hg rebase -s 14 -d 1 --keepbranches -q
$ hg branch -r 0: stable
changed branch on 6 changesets
$ hg glog
@ 23:6a5ddbcfb870 added bar
| stable (b1)
o 22:baedc6e98a67 Added e
| stable ()
| o 21:99ac7bf8aad1 Added d
| | stable ()
| o 20:0ecb4d39c4bd Added c
|/ stable ()
o 19:fd45b986b109 Added b
| stable ()
o 18:204d2769eca2 Added a
stable ()
$ hg branches
stable 23:6a5ddbcfb870
$ hg branch
stable
Changing to same branch is no-op
$ hg branch -r 19::21 stable
changed branch on 0 changesets
Changing branch name to existing branch name if the branch of parent of root of
revs is same as the new branch name
$ hg branch -r 20::21 bugfix
changed branch on 2 changesets
$ hg glog
o 25:714defe1cf34 Added d
| bugfix ()
o 24:98394def28fc Added c
| bugfix ()
| @ 23:6a5ddbcfb870 added bar
| | stable (b1)
| o 22:baedc6e98a67 Added e
|/ stable ()
o 19:fd45b986b109 Added b
| stable ()
o 18:204d2769eca2 Added a
stable ()
$ hg branch -r 24:25 stable
changed branch on 2 changesets
$ hg glog
o 27:4ec342341562 Added d
| stable ()
o 26:83f48859c2de Added c
| stable ()
| @ 23:6a5ddbcfb870 added bar
| | stable (b1)
| o 22:baedc6e98a67 Added e
|/ stable ()
o 19:fd45b986b109 Added b
| stable ()
o 18:204d2769eca2 Added a
stable ()
Changing branch of a merge commit
$ hg branch -q ghi
$ echo f > f
$ hg ci -qAm 'Added f'
$ hg up -q 27
$ hg branch -q jkl
$ echo g > g
$ hg ci -qAm 'Added g'
$ hg glog -r 'heads(:)'
@ 29:6bc1c6c2c9da Added g
| jkl ()
~
o 28:2f1019bd29d2 Added f
| ghi (b1)
~
$ hg branch -q default
$ hg merge -r 28
4 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ hg branch -r . abcd
abort: outstanding uncommitted merge
[255]
$ hg ci -m "Merge commit"
$ hg glog -r 'parents(.)::'
@ 30:4d56e6b1eb6b Merge commit
|\ default ()
| o 29:6bc1c6c2c9da Added g
| | jkl ()
| ~
o 28:2f1019bd29d2 Added f
| ghi (b1)
~
$ hg branch -r . ghi
0 files updated, 0 files merged, 4 files removed, 0 files unresolved
changed branch on 1 changesets
$ hg branch -r . jkl
changed branch on 1 changesets
$ hg branch -r . default
changed branch on 1 changesets
$ hg branch -r . stable
abort: a branch of the same name already exists
[255]
$ hg branch -r . stable --force
changed branch on 1 changesets
$ hg branches
stable 34:d1c2addda4a2
jkl 29:6bc1c6c2c9da (inactive)
ghi 28:2f1019bd29d2 (inactive)
Changing branch on public changeset
$ hg phase -r . -p
$ hg branch -r . def
abort: cannot change branch of public changesets
(see 'hg help phases' for details)
[255]
Merge commit with conflicts, with evolution and without
$ mklozenge() {
> echo foo > a
> hg ci -qAm foo
> echo bar > a
> hg ci -qm bar
> hg up -q '.^'
> echo baz > a
> hg ci -qm baz
> hg merge -q -t :local
> echo neither > a
> hg ci -qm neither
> }
$ cd ..
$ hg init merge-with-evolution
$ cd merge-with-evolution
$ mklozenge
$ hg branch -r '(.^)::' abc
changed branch on 2 changesets
$ hg glog
@ 5:c07fa8b34d54 neither
|\ abc ()
| o 4:f2aa51777cc9 baz
| | abc ()
o | 1:2e33c4f0856b bar
|/ default ()
o 0:91cfb6004abf foo
default ()
$ hg cat a
neither
$ cd ..
$ hg init merge-without-evolution
$ cd merge-without-evolution
$ mklozenge
$ cat > .hg/hgrc << EOF
> [experimental]
> evolution = no
> evolution.allowunstable = no
> EOF
$ hg branch -r '(.^)::' abc
changed branch on 2 changesets
saved backup bundle to $TESTTMP/merge-without-evolution/.hg/strip-backup/9a3a2af368f4-8db1a361-branch-change.hg
$ hg glog
@ 3:c07fa8b34d54 neither
|\ abc ()
| o 2:f2aa51777cc9 baz
| | abc ()
o | 1:2e33c4f0856b bar
|/ default ()
o 0:91cfb6004abf foo
default ()
$ hg cat a
neither