wireprotov2: move response handling out of httppeer
And fix some bugs while we're here.
The code for processing response data from the unified framing
protocol is mostly peer agnostic. The peer-specific bits are the
configuration of the client reactor and how I/O is performed. I
initially implemented things in httppeer for expediency.
This commit establishes a module for holding the peer API level
code for the framing based protocol. Inside this module we have
a class to help coordinate higher-level activities, such as managing
response object.
The client handler bits could be rolled into clientreactor. However,
I want clientreactor to be sans I/O and I want it to only be
concerned with protocol-level details, not higher-level concepts
like how protocol events are converted into peer API concepts. I
want clientreactor to receive a frame and then tell the caller what
should probably be done about it. If we start putting things like
future resolution into clientreactor, we'll constrain how the protocol
can be used (e.g. by requiring futures).
The new code is loosely based on what was in httppeer before. I
changed things a bit around response handling. We now buffer the
entire response "body" and then handle it as one atomic unit. This
fixed a bug around decoding CBOR data that spanned multiple frames.
I also fixed an off-by-one bug where we failed to read a single byte
CBOR value at the end of the stream. That's why tests have changed.
The new state of httppeer is much cleaner. It is largely agnostic
about framing protocol implementation details. That's how it should
be: the framing protocol is designed to be largely transport
agnostic. We want peers merely putting bytes on the wire and telling
the framing protocol where to read response data from.
There's still a bit of work to be done here, especially for
representing responses. But at least we're a step closer to having a
higher-level peer interface that can be plugged into the SSH peer
someday.
I initially added this class to wireprotoframing. However, we'll
eventually need version 2 specific functions to convert CBOR responses
into data structures expected by the code calling commands. This
needs to live somewhere. Since that code would be shared across peers,
we need a common module. We have wireprotov1peer for the equivalent
version 1 code. So I decided to establish wireprotov2peer.
Differential Revision: https://phab.mercurial-scm.org/D3379
$ . "$TESTDIR/narrow-library.sh"
$ hg init master
$ cd master
$ mkdir inside
$ echo inside > inside/f1
$ mkdir outside
$ echo outside > outside/f2
$ mkdir patchdir
$ echo patch_this > patchdir/f3
$ hg ci -Aqm 'initial'
$ cd ..
$ hg clone --narrow ssh://user@dummy/master narrow --include inside
requesting all changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
new changesets dff6a2a6d433
updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd narrow
$ mkdir outside
$ echo other_contents > outside/f2
$ grep outside .hg/narrowspec
[1]
$ grep outside .hg/dirstate
[1]
$ hg status
`hg status` did not add outside.
$ grep outside .hg/narrowspec
[1]
$ grep outside .hg/dirstate
[1]
Unfortunately this is not really a candidate for adding to narrowhg proper,
since it depends on some other source for providing the manifests (when using
treemanifests) and file contents. Something like a virtual filesystem and/or
remotefilelog. We want to be useful when not using those systems, so we do not
have this method available in narrowhg proper at the moment.
$ cat > "$TESTTMP/expand_extension.py" <<EOF
> import os
> import sys
>
> from mercurial import encoding
> from mercurial import extensions
> from mercurial import localrepo
> from mercurial import match as matchmod
> from mercurial import narrowspec
> from mercurial import patch
> from mercurial import util as hgutil
>
> def expandnarrowspec(ui, repo, newincludes=None):
> if not newincludes:
> return
> import sys
> newincludes = set([newincludes])
> includes, excludes = repo.narrowpats
> currentmatcher = narrowspec.match(repo.root, includes, excludes)
> includes = includes | newincludes
> if not repo.currenttransaction():
> ui.develwarn(b'expandnarrowspec called outside of transaction!')
> repo.setnarrowpats(includes, excludes)
> newmatcher = narrowspec.match(repo.root, includes, excludes)
> added = matchmod.differencematcher(newmatcher, currentmatcher)
> for f in repo[b'.'].manifest().walk(added):
> repo.dirstate.normallookup(f)
>
> def makeds(ui, repo):
> def wrapds(orig, self):
> ds = orig(self)
> class expandingdirstate(ds.__class__):
> @hgutil.propertycache
> def _map(self):
> ret = super(expandingdirstate, self)._map
> with repo.wlock(), repo.lock(), repo.transaction(
> b'expandnarrowspec'):
> expandnarrowspec(ui, repo,
> encoding.environ.get(b'DIRSTATEINCLUDES'))
> return ret
> ds.__class__ = expandingdirstate
> return ds
> return wrapds
>
> def reposetup(ui, repo):
> extensions.wrapfilecache(localrepo.localrepository, b'dirstate',
> makeds(ui, repo))
> def overridepatch(orig, *args, **kwargs):
> with repo.wlock():
> expandnarrowspec(ui, repo, encoding.environ.get(b'PATCHINCLUDES'))
> return orig(*args, **kwargs)
>
> extensions.wrapfunction(patch, b'patch', overridepatch)
> EOF
$ cat >> ".hg/hgrc" <<EOF
> [extensions]
> expand_extension = $TESTTMP/expand_extension.py
> EOF
Since we do not have the ability to rely on a virtual filesystem or
remotefilelog in the test, we just fake it by copying the data from the 'master'
repo.
$ cp -a ../master/.hg/store/data/* .hg/store/data
Do that for patchdir as well.
$ cp -a ../master/patchdir .
`hg status` will now add outside, but not patchdir.
$ DIRSTATEINCLUDES=path:outside hg status
M outside/f2
$ grep outside .hg/narrowspec
path:outside
$ grep outside .hg/dirstate > /dev/null
$ grep patchdir .hg/narrowspec
[1]
$ grep patchdir .hg/dirstate
[1]
Get rid of the modification to outside/f2.
$ hg update -C .
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
This patch will not apply cleanly at the moment, so `hg import` will break
$ cat > "$TESTTMP/foo.patch" <<EOF
> --- patchdir/f3
> +++ patchdir/f3
> @@ -1,1 +1,1 @@
> -this should be "patch_this", but its not, so patch fails
> +this text is irrelevant
> EOF
$ PATCHINCLUDES=path:patchdir hg import -p0 -e "$TESTTMP/foo.patch" -m ignored
applying $TESTTMP/foo.patch
patching file patchdir/f3
Hunk #1 FAILED at 0
1 out of 1 hunks FAILED -- saving rejects to file patchdir/f3.rej
abort: patch failed to apply
[255]
$ grep patchdir .hg/narrowspec
[1]
$ grep patchdir .hg/dirstate > /dev/null
[1]
Let's make it apply cleanly and see that it *did* expand properly
$ cat > "$TESTTMP/foo.patch" <<EOF
> --- patchdir/f3
> +++ patchdir/f3
> @@ -1,1 +1,1 @@
> -patch_this
> +patched_this
> EOF
$ PATCHINCLUDES=path:patchdir hg import -p0 -e "$TESTTMP/foo.patch" -m message
applying $TESTTMP/foo.patch
$ cat patchdir/f3
patched_this
$ grep patchdir .hg/narrowspec
path:patchdir
$ grep patchdir .hg/dirstate > /dev/null