wireproto: implement batching on peer executor interface
This is a bit more complicated than non-batch requests because we
need to buffer sends until the last request arrives *and* we need
to support resolving futures as data arrives from the remote.
In a classical concurrent.futures executor model, the future
"starts" as soon as it is submitted. However, we have nothing to
start until the last command is submitted.
If we did nothing, calling result() would deadlock, since the future
hasn't "started." So in the case where we queue the command, we return
a special future type whose result() will trigger sendcommands().
This eliminates the deadlock potential. It also serves as a check
against callers who may be calling result() prematurely, as it will
prevent any subsequent callcommands() from working. This behavior
is slightly annoying and a bit restrictive. But it's the world
that half duplex connections forces on us.
In order to support streaming responses, we were previously using
a generator. But with a futures-based API, we're using futures
and not generators. So in order to get streaming, we need a
background thread to read data from the server.
The approach taken in this patch is to leverage the ThreadPoolExecutor
from concurrent.futures for managing a background thread. We create
an executor and future that resolves when all response data is
processed (or an error occurs). When exiting the context manager,
we wait on that background reading before returning.
I was hoping we could manually spin up a threading.Thread and this
would be simple. But I ran into a few deadlocks when implementing.
After looking at the source code to concurrent.futures, I figured
it would just be easier to use a ThreadPoolExecutor than implement
all the code needed to manually manage a thread.
To prove this works, a use of the batch API in discovery has been
updated.
Differential Revision: https://phab.mercurial-scm.org/D3269
$ cat >> $HGRCPATH << EOF
> [diff]
> git = true
> EOF
$ hg init
$ cat > foo << EOF
> 0
> 1
> 2
> 3
> 4
> EOF
$ hg ci -Am init
adding foo
$ cat > foo << EOF
> 0
> 0
> 0
> 0
> 1
> 2
> 3
> 4
> EOF
$ hg ci -m 'more 0'
$ sed 's/2/2+/' foo > foo.new
$ mv foo.new foo
$ cat > bar << EOF
> a
> b
> c
> d
> e
> EOF
$ hg add bar
$ hg ci -Am "2 -> 2+; added bar"
$ cat >> foo << EOF
> 5
> 6
> 7
> 8
> 9
> 10
> 11
> EOF
$ hg ci -m "to 11"
Add some changes with two diff hunks
$ sed 's/^1$/ 1/' foo > foo.new
$ mv foo.new foo
$ sed 's/^11$/11+/' foo > foo.new
$ mv foo.new foo
$ hg ci -m '11 -> 11+; leading space before "1"'
(make sure there are two hunks in "foo")
$ hg diff -c .
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -2,7 +2,7 @@
0
0
0
-1
+ 1
2+
3
4
@@ -12,4 +12,4 @@
8
9
10
-11
+11+
$ sed 's/3/3+/' foo > foo.new
$ mv foo.new foo
$ sed 's/^11+$/11-/' foo > foo.new
$ mv foo.new foo
$ sed 's/a/a+/' bar > bar.new
$ mv bar.new bar
$ hg ci -m 'foo: 3 -> 3+ and 11+ -> 11-; bar: a -> a+'
(make sure there are two hunks in "foo")
$ hg diff -c . foo
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -4,7 +4,7 @@
0
1
2+
-3
+3+
4
5
6
@@ -12,4 +12,4 @@
8
9
10
-11+
+11-
$ hg log -f -L foo,5:7 -p
changeset: 5:cfdf972b3971
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: foo: 3 -> 3+ and 11+ -> 11-; bar: a -> a+
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -4,7 +4,7 @@
0
1
2+
-3
+3+
4
5
6
changeset: 4:eaec41c1a0c9
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: 11 -> 11+; leading space before "1"
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -2,7 +2,7 @@
0
0
0
-1
+ 1
2+
3
4
changeset: 2:63a884426fd0
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: 2 -> 2+; added bar
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -3,6 +3,6 @@
0
0
1
-2
+2+
3
4
changeset: 0:5ae1f82b9a00
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: init
diff --git a/foo b/foo
new file mode 100644
--- /dev/null
+++ b/foo
@@ -0,0 +1,5 @@
+0
+1
+2
+3
+4
$ hg log -f --graph -L foo,5:7 -p
@ changeset: 5:cfdf972b3971
| tag: tip
| user: test
| date: Thu Jan 01 00:00:00 1970 +0000
| summary: foo: 3 -> 3+ and 11+ -> 11-; bar: a -> a+
|
| diff --git a/foo b/foo
| --- a/foo
| +++ b/foo
| @@ -4,7 +4,7 @@
| 0
| 1
| 2+
| -3
| +3+
| 4
| 5
| 6
|
o changeset: 4:eaec41c1a0c9
: user: test
: date: Thu Jan 01 00:00:00 1970 +0000
: summary: 11 -> 11+; leading space before "1"
:
: diff --git a/foo b/foo
: --- a/foo
: +++ b/foo
: @@ -2,7 +2,7 @@
: 0
: 0
: 0
: -1
: + 1
: 2+
: 3
: 4
:
o changeset: 2:63a884426fd0
: user: test
: date: Thu Jan 01 00:00:00 1970 +0000
: summary: 2 -> 2+; added bar
:
: diff --git a/foo b/foo
: --- a/foo
: +++ b/foo
: @@ -3,6 +3,6 @@
: 0
: 0
: 1
: -2
: +2+
: 3
: 4
:
o changeset: 0:5ae1f82b9a00
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: init
diff --git a/foo b/foo
new file mode 100644
--- /dev/null
+++ b/foo
@@ -0,0 +1,5 @@
+0
+1
+2
+3
+4
With --template.
$ hg log -f -L foo,5:7 -T '{rev}:{node|short} {desc|firstline}\n'
5:cfdf972b3971 foo: 3 -> 3+ and 11+ -> 11-; bar: a -> a+
4:eaec41c1a0c9 11 -> 11+; leading space before "1"
2:63a884426fd0 2 -> 2+; added bar
0:5ae1f82b9a00 init
$ hg log -f -L foo,5:7 -T json
[
{
"rev": 5,
"node": "cfdf972b3971a2a59638bf9583c0debbffee5404",
"branch": "default",
"phase": "draft",
"user": "test",
"date": [0, 0],
"desc": "foo: 3 -> 3+ and 11+ -> 11-; bar: a -> a+",
"bookmarks": [],
"tags": ["tip"],
"parents": ["eaec41c1a0c9ad0a5e999611d0149d171beffb8c"]
},
{
"rev": 4,
"node": "eaec41c1a0c9ad0a5e999611d0149d171beffb8c",
"branch": "default",
"phase": "draft",
"user": "test",
"date": [0, 0],
"desc": "11 -> 11+; leading space before \"1\"",
"bookmarks": [],
"tags": [],
"parents": ["730a61fbaecf426c17c2c66bc42d195b5d5b0ba8"]
},
{
"rev": 2,
"node": "63a884426fd0b277fcd55895bbb2f230434576eb",
"branch": "default",
"phase": "draft",
"user": "test",
"date": [0, 0],
"desc": "2 -> 2+; added bar",
"bookmarks": [],
"tags": [],
"parents": ["29a1e7c6b80024f63f310a2d71de979e9d2996d7"]
},
{
"rev": 0,
"node": "5ae1f82b9a000ff1e0967d0dac1c58b9d796e1b4",
"branch": "default",
"phase": "draft",
"user": "test",
"date": [0, 0],
"desc": "init",
"bookmarks": [],
"tags": [],
"parents": ["0000000000000000000000000000000000000000"]
}
]
With some white-space diff option, respective revisions are skipped.
$ hg log -f -L foo,5:7 -p --config diff.ignorews=true
changeset: 5:cfdf972b3971
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: foo: 3 -> 3+ and 11+ -> 11-; bar: a -> a+
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -4,7 +4,7 @@
0
1
2+
-3
+3+
4
5
6
changeset: 2:63a884426fd0
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: 2 -> 2+; added bar
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -3,6 +3,6 @@
0
0
1
-2
+2+
3
4
changeset: 0:5ae1f82b9a00
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: init
diff --git a/foo b/foo
new file mode 100644
--- /dev/null
+++ b/foo
@@ -0,0 +1,5 @@
+0
+1
+2
+3
+4
Regular file patterns are not allowed.
$ hg log -f -L foo,5:7 -p bar
abort: FILE arguments are not compatible with --line-range option
[255]
Option --rev acts as a restriction.
$ hg log -f -L foo,5:7 -p -r 'desc(2)'
changeset: 2:63a884426fd0
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: 2 -> 2+; added bar
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -3,6 +3,6 @@
0
0
1
-2
+2+
3
4
changeset: 0:5ae1f82b9a00
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: init
diff --git a/foo b/foo
new file mode 100644
--- /dev/null
+++ b/foo
@@ -0,0 +1,5 @@
+0
+1
+2
+3
+4
With several -L patterns, changes touching any files in their respective line
range are show.
$ hg log -f -L foo,5:7 -L bar,1:2 -p
changeset: 5:cfdf972b3971
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: foo: 3 -> 3+ and 11+ -> 11-; bar: a -> a+
diff --git a/bar b/bar
--- a/bar
+++ b/bar
@@ -1,4 +1,4 @@
-a
+a+
b
c
d
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -4,7 +4,7 @@
0
1
2+
-3
+3+
4
5
6
changeset: 4:eaec41c1a0c9
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: 11 -> 11+; leading space before "1"
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -2,7 +2,7 @@
0
0
0
-1
+ 1
2+
3
4
changeset: 2:63a884426fd0
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: 2 -> 2+; added bar
diff --git a/bar b/bar
new file mode 100644
--- /dev/null
+++ b/bar
@@ -0,0 +1,5 @@
+a
+b
+c
+d
+e
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -3,6 +3,6 @@
0
0
1
-2
+2+
3
4
changeset: 0:5ae1f82b9a00
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: init
diff --git a/foo b/foo
new file mode 100644
--- /dev/null
+++ b/foo
@@ -0,0 +1,5 @@
+0
+1
+2
+3
+4
Multiple -L options with the same file yields changes touching any of
specified line ranges.
$ hg log -f -L foo,5:7 -L foo,14:15 -p
changeset: 5:cfdf972b3971
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: foo: 3 -> 3+ and 11+ -> 11-; bar: a -> a+
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -4,7 +4,7 @@
0
1
2+
-3
+3+
4
5
6
@@ -12,4 +12,4 @@
8
9
10
-11+
+11-
changeset: 4:eaec41c1a0c9
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: 11 -> 11+; leading space before "1"
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -2,7 +2,7 @@
0
0
0
-1
+ 1
2+
3
4
@@ -12,4 +12,4 @@
8
9
10
-11
+11+
changeset: 3:730a61fbaecf
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: to 11
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -6,3 +6,10 @@
2+
3
4
+5
+6
+7
+8
+9
+10
+11
changeset: 2:63a884426fd0
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: 2 -> 2+; added bar
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -3,6 +3,6 @@
0
0
1
-2
+2+
3
4
changeset: 0:5ae1f82b9a00
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: init
diff --git a/foo b/foo
new file mode 100644
--- /dev/null
+++ b/foo
@@ -0,0 +1,5 @@
+0
+1
+2
+3
+4
A file with a comma in its name.
$ cat > ba,z << EOF
> q
> w
> e
> r
> t
> y
> EOF
$ hg ci -Am 'querty'
adding ba,z
$ cat >> ba,z << EOF
> u
> i
> o
> p
> EOF
$ hg ci -m 'more keys'
$ cat > ba,z << EOF
> a
> z
> e
> r
> t
> y
> u
> i
> o
> p
> EOF
$ hg ci -m 'azerty'
$ hg log -f -L ba,z,1:2 -p
changeset: 8:52373265138b
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: azerty
diff --git a/ba,z b/ba,z
--- a/ba,z
+++ b/ba,z
@@ -1,5 +1,5 @@
-q
-w
+a
+z
e
r
t
changeset: 6:96ba8850f316
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: querty
diff --git a/ba,z b/ba,z
new file mode 100644
--- /dev/null
+++ b/ba,z
@@ -0,0 +1,6 @@
+q
+w
+e
+r
+t
+y
Exact prefix kinds work in -L options.
$ mkdir dir
$ cd dir
$ hg log -f -L path:foo,5:7 -p
changeset: 5:cfdf972b3971
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: foo: 3 -> 3+ and 11+ -> 11-; bar: a -> a+
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -4,7 +4,7 @@
0
1
2+
-3
+3+
4
5
6
changeset: 4:eaec41c1a0c9
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: 11 -> 11+; leading space before "1"
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -2,7 +2,7 @@
0
0
0
-1
+ 1
2+
3
4
changeset: 2:63a884426fd0
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: 2 -> 2+; added bar
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -3,6 +3,6 @@
0
0
1
-2
+2+
3
4
changeset: 0:5ae1f82b9a00
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: init
diff --git a/foo b/foo
new file mode 100644
--- /dev/null
+++ b/foo
@@ -0,0 +1,5 @@
+0
+1
+2
+3
+4
Renames are followed.
$ hg mv ../foo baz
$ sed 's/1/1+/' baz > baz.new
$ mv baz.new baz
$ hg ci -m 'foo -> dir/baz; 1-1+'
$ hg diff -c .
diff --git a/foo b/dir/baz
rename from foo
rename to dir/baz
--- a/foo
+++ b/dir/baz
@@ -2,7 +2,7 @@
0
0
0
- 1
+ 1+
2+
3+
4
@@ -11,5 +11,5 @@
7
8
9
-10
-11-
+1+0
+1+1-
$ hg log -f -L relpath:baz,5:7 -p
changeset: 9:6af29c3a778f
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: foo -> dir/baz; 1-1+
diff --git a/foo b/dir/baz
copy from foo
copy to dir/baz
--- a/foo
+++ b/dir/baz
@@ -2,7 +2,7 @@
0
0
0
- 1
+ 1+
2+
3+
4
changeset: 5:cfdf972b3971
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: foo: 3 -> 3+ and 11+ -> 11-; bar: a -> a+
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -4,7 +4,7 @@
0
1
2+
-3
+3+
4
5
6
changeset: 4:eaec41c1a0c9
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: 11 -> 11+; leading space before "1"
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -2,7 +2,7 @@
0
0
0
-1
+ 1
2+
3
4
changeset: 2:63a884426fd0
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: 2 -> 2+; added bar
diff --git a/foo b/foo
--- a/foo
+++ b/foo
@@ -3,6 +3,6 @@
0
0
1
-2
+2+
3
4
changeset: 0:5ae1f82b9a00
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: init
diff --git a/foo b/foo
new file mode 100644
--- /dev/null
+++ b/foo
@@ -0,0 +1,5 @@
+0
+1
+2
+3
+4
Binary files work but without diff hunks filtering.
(Checking w/ and w/o diff.git option.)
>>> open('binary', 'wb').write(b'this\nis\na\nbinary\0') and None
$ hg add binary
$ hg ci -m 'add a binary file' --quiet
$ hg log -f -L binary,1:2 -p
changeset: 10:c96381c229df
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: add a binary file
diff --git a/dir/binary b/dir/binary
new file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..c2e1fbed209fe919b3f189a6a31950e9adf61e45
GIT binary patch
literal 17
Wc$_QA$SmdpqC~Ew%)G>+N(KNlNClYy
$ hg log -f -L binary,1:2 -p --config diff.git=false
changeset: 10:c96381c229df
tag: tip
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: add a binary file
diff -r 6af29c3a778f -r c96381c229df dir/binary
Binary file dir/binary has changed
Option --follow is required.
$ hg log -L foo,5:7
abort: --line-range requires --follow
[255]
Non-exact pattern kinds are not allowed.
$ cd ..
$ hg log -f -L glob:*a*,1:2
hg: parse error: line range pattern 'glob:*a*' must match exactly one file
[255]
We get an error for removed files.
$ hg rm dir/baz
$ hg ci -m 'remove baz' --quiet
$ hg log -f -L dir/baz,5:7 -p
abort: cannot follow file not in parent revision: "dir/baz"
[255]