tests/test-mq

named branches: server branchmap wire protocol support (issue736)

The repository command, 'branchmap', returns a dictionary, branchname
-> [branchheads], and will be implemented for localrepo, httprepo and
sshrepo.

The following wire format is used for returning data:

branchname1 branch1head2 branch1head2 ...
branchname2 ...
...

Branch names are URL encoded to escape white space, and branch heads
are sent as hex encoded node ids. All branches and all their heads are
sent.

The background and motivation for this command is the desire for a
richer named branch semantics when pushing changesets. The details are
explained in the original proposal which is included below.


1. BACKGROUND

The algorithm currently implemented in Mercurial only considers the
graph theoretical heads when determining whether new heads are
created, rather than using the branch heads as a count (the algorithm
considers a branch head effectively closed when it is merged into
another branch or a new named branch is started from that point
onward).

Our particular problem with the algorithm is that we'd like to see the
following case working without forcing a push:

Upsteam has:

(0:dev) ---- (1:dev)
\
 `--- (2:stable)

Someone merges stable into dev:

(0:dev) ---- (1:dev) ------(3:dev)
\                         /
 `--- (2:stable) --------´

This can be pushed without --force (as it should).
Now someone else does some coding on stable (a bug fix, say):

(0:dev) ---- (1:dev) ------(3:dev)
\                          /
 `--- (2:stable) ---------´---------(4:stable)

This time we need --force to push.

We allow this to be pushed without using --force by getting all the
remote branch heads (by extending the wire protocol with a new
function).

We would, furthermore, also prefer if it is impossible to push a new
branch without --force (or a later --newbranch option so --force isn't
shoe-horned into too many disparate functions, if need be), except of
course in the case where the remote repository is empty.

This is what our patches accomplish.


2. ALTERNATIVES

We have, of course, considered some alternatives to reconstructing
enough information to decide whether we are creating new remote branch
heads, before we added the new wire protocol command.

2.1. LOOKUP ON REMOTE

The main alternative is to use the information from remote.heads() and
remote.lookup() to try to reconstruct enough graph information to
decide whether we are creating new heads. This is not adequate as
illustrated below.

Remember that each lookup is typically a request-response pair over
SSH or HTTP(S).

If we have a simple repository at the remote end like this:

(0:dev) ---- (1:dev) ---- (3:stable)
\
 `--- (2:dev)

then remote.heads() will yield [2, 3]. Assume we have nodes [0, 1, 2]
locally and want to create a new node, 4:dev, as a descendant from
(1:dev), which should be OK as 1:dev is a branch head.

If we do remote.lookup('dev') we will get [2]. Thus, we can get
information about whether a branch exists on the remote server or not,
but this does not solve our problem of figuring out whether we are
creating new heads or not.

Pushing 4:dev ought to be OK, since after the push, we still only have
two heads on branch a.

Using remote.lookup() and remote.heads() is thus not adequate to
consistently decide whether we are creating new remote heads (e.g. in
this situation the latter would never return 1:dev).

2.2. USING INCOMING TO RECONSTRUCT THE GRAPH

An alternative would be to use information equivalent to hg incoming
to get the full remote graph in addition to the local graph.

To do this, we would have to get a changegroup(subset) bundle
representing the remote end (which may be a substantial amount of
data), getting the branch heads from an instantiated bundlerepository,
deleting the bundle, and finally, we can compute the prepush logic.

While this is backwards compatible, it will cause a possibly
substantial slowdown of the push command as it first needs to pull in
all changes.


3. FURTHER ARGUMENTS IN FAVOUR OF THE BRANCHMAP WIRE-PROTOCOL EXTENSION

Currently, the commands incoming and pull, work based on the tip of a
given branch if used with "-r branchname", making it hard to get all
revisions of a certain branch only (if it has multiple heads). This
can be solved by requesting the remote's branchheads and letting the
revisions to be used with the command be these heads. This can be done
by extending the commands with a new option, e.g.:

hg pull -b branchname

which will be turned into the equivalent of:

hg pull -r branchhead1 -r branchhead2 -r branchhead3

We have a simple follow-up patch that can do this ready as well
(although not submitted yet as it is pending the acceptance of the
branch patch).


4. WRAP-UP

We generally find that the branchmap wire protocol extension can
provide better named branch support to Mercurial. Currently, some
things, like the initial push scenario in this mail, are fairly
counter-intuitive, and the more often you have to force push, the more
it is likely you will get a lot of spurious and unnecessary merge
nodes. Also, restricting incoming and pull to all changes on a branch
rather than changes on the tip-most head would be a sensible extension
to making named branches a first class citizen in Mercurial.
Currently, named branches sometimes feel like a late-coming unwanted
step-child.

We have run it in a production environment for a while, with fewer
multiple heads occurring in our repositories and fewer confused users
as a result.

Also, it fixes the long-standing issue 736.

Co-contributor: Sune Foldager <[e-mail]>

#!/bin/sh

checkundo()
{
    if [ -f .hg/store/undo ]; then
	echo ".hg/store/undo still exists after $1"
    fi
}

echo "[extensions]" >> $HGRCPATH
echo "mq=" >> $HGRCPATH

echo % help
hg help mq

hg init a
cd a
echo a > a
hg ci -Ama

hg clone . ../k

mkdir b
echo z > b/z
hg ci -Ama

echo % qinit

hg qinit

cd ..
hg init b

echo % -R qinit

hg -R b qinit

hg init c

echo % qinit -c

hg --cwd c qinit -c
hg -R c/.hg/patches st

echo '% qinit; qinit -c'
hg init d
cd d
hg qinit
hg qinit -c
# qinit -c should create both files if they don't exist
echo '  .hgignore:'
cat .hg/patches/.hgignore
echo '  series:'
cat .hg/patches/series
hg qinit -c 2>&1 | sed -e 's/repository.*already/repository already/'
cd ..

echo '% qinit; <stuff>; qinit -c'
hg init e
cd e
hg qnew A
checkundo qnew
echo foo > foo
hg add foo
hg qrefresh
hg qnew B
echo >> foo
hg qrefresh
echo status >> .hg/patches/.hgignore
echo bleh >> .hg/patches/.hgignore
hg qinit -c
hg -R .hg/patches status
# qinit -c shouldn't touch these files if they already exist
echo '  .hgignore:'
cat .hg/patches/.hgignore
echo '  series:'
cat .hg/patches/series
cd ..

cd a

hg qnew -m 'foo bar' test.patch

echo % qrefresh

echo a >> a
hg qrefresh
sed -e "s/^\(diff -r \)\([a-f0-9]* \)/\1 x/" \
    -e "s/\(+++ [a-zA-Z0-9_/.-]*\).*/\1/" \
    -e "s/\(--- [a-zA-Z0-9_/.-]*\).*/\1/" .hg/patches/test.patch

echo % empty qrefresh

hg qrefresh -X a
echo 'revision:'
hg diff -r -2 -r -1
echo 'patch:'
cat .hg/patches/test.patch
echo 'working dir diff:'
hg diff --nodates -q
# restore things
hg qrefresh
checkundo qrefresh

echo % qpop

hg qpop
checkundo qpop

echo % qpush

hg qpush
checkundo qpush

cd ..

echo % pop/push outside repo

hg -R a qpop
hg -R a qpush

cd a
hg qnew test2.patch

echo % qrefresh in subdir

cd b
echo a > a
hg add a
hg qrefresh

echo % pop/push -a in subdir

hg qpop -a
hg --traceback qpush -a

echo % qseries
hg qseries
hg qpop
hg qseries -vs
hg qpush

echo % qapplied
hg qapplied

echo % qtop
hg qtop

echo % qprev
hg qprev

echo % qnext
hg qnext

echo % pop, qnext, qprev, qapplied
hg qpop
hg qnext
hg qprev
hg qapplied

echo % commit should fail
hg commit

echo % push should fail
hg push ../../k

echo % import should fail
hg st .
echo foo >> ../a
hg diff > ../../import.diff
hg revert --no-backup ../a
hg import ../../import.diff
hg st

echo % qunapplied
hg qunapplied

echo % qpush/qpop with index
hg qnew test1b.patch
echo 1b > 1b
hg add 1b
hg qrefresh
hg qpush 2
hg qpop 0
hg qpush test.patch+1
hg qpush test.patch+2
hg qpop test2.patch-1
hg qpop test2.patch-2
hg qpush test1b.patch+1

echo % push should succeed
hg qpop -a
hg push ../../k

echo % qpush/qpop error codes
errorcode()
{
    hg "$@" && echo "  $@ succeeds" || echo "  $@ fails"
}

# we want to start with some patches applied
hg qpush -a
echo "  % pops all patches and succeeds"
errorcode qpop -a
echo "  % does nothing and succeeds"
errorcode qpop -a
echo "  % fails - nothing else to pop"
errorcode qpop
echo "  % pushes a patch and succeeds"
errorcode qpush
echo "  % pops a patch and succeeds"
errorcode qpop
echo "  % pushes up to test1b.patch and succeeds"
errorcode qpush test1b.patch
echo "  % does nothing and succeeds"
errorcode qpush test1b.patch
echo "  % does nothing and succeeds"
errorcode qpop test1b.patch
echo "  % fails - can't push to this patch"
errorcode qpush test.patch
echo "  % fails - can't pop to this patch"
errorcode qpop test2.patch
echo "  % pops up to test.patch and succeeds"
errorcode qpop test.patch
echo "  % pushes all patches and succeeds"
errorcode qpush -a
echo "  % does nothing and succeeds"
errorcode qpush -a
echo "  % fails - nothing else to push"
errorcode qpush
echo "  % does nothing and succeeds"
errorcode qpush test2.patch


echo % strip
cd ../../b
echo x>x
hg ci -Ama
hg strip tip 2>&1 | sed 's/\(saving bundle to \).*/\1/'
hg unbundle .hg/strip-backup/*

echo % strip with local changes, should complain
hg up
echo y>y
hg add y
hg strip tip | sed 's/\(saving bundle to \).*/\1/'
echo % --force strip with local changes
hg strip -f tip 2>&1 | sed 's/\(saving bundle to \).*/\1/'

echo '% cd b; hg qrefresh'
hg init refresh
cd refresh
echo a > a
hg ci -Ama
hg qnew -mfoo foo
echo a >> a
hg qrefresh
mkdir b
cd b
echo f > f
hg add f
hg qrefresh
sed -e "s/\(+++ [a-zA-Z0-9_/.-]*\).*/\1/" \
    -e "s/\(--- [a-zA-Z0-9_/.-]*\).*/\1/" ../.hg/patches/foo
echo % hg qrefresh .
hg qrefresh .
sed -e "s/\(+++ [a-zA-Z0-9_/.-]*\).*/\1/" \
    -e "s/\(--- [a-zA-Z0-9_/.-]*\).*/\1/" ../.hg/patches/foo
hg status

echo % qpush failure
cd ..
hg qrefresh
hg qnew -mbar bar
echo foo > foo
echo bar > bar
hg add foo bar
hg qrefresh
hg qpop -a
echo bar > foo
hg qpush -a
hg st

echo % mq tags
hg log --template '{rev} {tags}\n' -r qparent:qtip

echo % bad node in status
hg qpop
hg strip -qn tip
hg tip 2>&1 | sed -e 's/unknown node .*/unknown node/'
hg branches 2>&1 | sed -e 's/unknown node .*/unknown node/'
hg qpop 2>&1 | sed -e 's/unknown node .*/unknown node/'

cat >>$HGRCPATH <<EOF
[diff]
git = True
EOF
cd ..
hg init git
cd git
hg qinit

hg qnew -m'new file' new
echo foo > new
chmod +x new
hg add new
hg qrefresh
sed -e "s/\(+++ [a-zA-Z0-9_/.-]*\).*/\1/" \
    -e "s/\(--- [a-zA-Z0-9_/.-]*\).*/\1/" .hg/patches/new

hg qnew -m'copy file' copy
hg cp new copy
hg qrefresh
sed -e "s/\(+++ [a-zA-Z0-9_/.-]*\).*/\1/" \
    -e "s/\(--- [a-zA-Z0-9_/.-]*\).*/\1/" .hg/patches/copy

hg qpop
hg qpush
hg qdiff
cat >>$HGRCPATH <<EOF
[diff]
git = False
EOF
hg qdiff --git

cd ..
hg init slow
cd slow
hg qinit
echo foo > foo
hg add foo
hg ci -m 'add foo'
hg qnew bar
echo bar > bar
hg add bar
hg mv foo baz
hg qrefresh --git
hg up -C 0
echo >> foo
hg ci -m 'change foo'
hg up -C 1
hg qrefresh --git 2>&1 | grep -v 'saving bundle'
cat .hg/patches/bar
hg log -vC --template '{rev} {file_copies%filecopy}\n' -r .
hg qrefresh --git
cat .hg/patches/bar
hg log -vC --template '{rev} {file_copies%filecopy}\n' -r .
hg qrefresh
grep 'diff --git' .hg/patches/bar

echo
hg up -C 1
echo >> foo
hg ci -m 'change foo again'
hg up -C 2
hg mv bar quux
hg mv baz bleh
hg qrefresh --git 2>&1 | grep -v 'saving bundle'
cat .hg/patches/bar
hg log -vC --template '{rev} {file_copies%filecopy}\n' -r .
hg mv quux fred
hg mv bleh barney
hg qrefresh --git
cat .hg/patches/bar
hg log -vC --template '{rev} {file_copies%filecopy}\n' -r .

echo % refresh omitting an added file
hg qnew baz
echo newfile > newfile
hg add newfile
hg qrefresh
hg st -A newfile
hg qrefresh -X newfile
hg st -A newfile
hg revert newfile
rm newfile
hg qpop
hg qdel baz

echo % create a git patch
echo a > alexander
hg add alexander
hg qnew -f --git addalexander
grep diff .hg/patches/addalexander

echo % create a git binary patch
cat > writebin.py <<EOF
import sys
path = sys.argv[1]
open(path, 'wb').write('BIN\x00ARY')
EOF
python writebin.py bucephalus

python "$TESTDIR/md5sum.py" bucephalus
hg add bucephalus
hg qnew -f --git addbucephalus
grep diff .hg/patches/addbucephalus

echo % check binary patches can be popped and pushed
hg qpop
test -f bucephalus && echo % bucephalus should not be there
hg qpush
test -f bucephalus || echo % bucephalus should be there
python "$TESTDIR/md5sum.py" bucephalus


echo '% strip again'
cd ..
hg init strip
cd strip
touch foo
hg add foo
hg ci -m 'add foo'
echo >> foo
hg ci -m 'change foo 1'
hg up -C 0
echo 1 >> foo
hg ci -m 'change foo 2'
HGMERGE=true hg merge
hg ci -m merge
hg log
hg strip 1 2>&1 | sed 's/\(saving bundle to \).*/\1/'
checkundo strip
hg log
cd ..

echo '% qclone'
qlog()
{
    echo 'main repo:'
    hg log --template '    rev {rev}: {desc}\n'
    echo 'patch repo:'
    hg -R .hg/patches log --template '    rev {rev}: {desc}\n'
}
hg init qclonesource
cd qclonesource
echo foo > foo
hg add foo
hg ci -m 'add foo'
hg qinit
hg qnew patch1
echo bar >> foo
hg qrefresh -m 'change foo'
cd ..

# repo with unversioned patch dir
hg qclone qclonesource failure

cd qclonesource
hg qinit -c
hg qci -m checkpoint
qlog
cd ..

# repo with patches applied
hg qclone qclonesource qclonedest
cd qclonedest
qlog
cd ..

# repo with patches unapplied
cd qclonesource
hg qpop -a
qlog
cd ..
hg qclone qclonesource qclonedest2
cd qclonedest2
qlog
cd ..

echo % 'test applying on an empty file (issue 1033)'
hg init empty
cd empty
touch a
hg ci -Am addempty
echo a > a
hg qnew -f -e changea
hg qpop
hg qpush
cd ..

echo % test qpush with --force, issue1087
hg init forcepush
cd forcepush
echo hello > hello.txt
echo bye > bye.txt
hg ci -Ama
hg qnew -d '0 0' empty
hg qpop
echo world >> hello.txt

echo % qpush should fail, local changes
hg qpush

echo % apply force, should not discard changes with empty patch
hg qpush -f 2>&1 | sed 's,^.*/patch,patch,g'
hg diff --config diff.nodates=True
hg qdiff --config diff.nodates=True
hg log -l1 -p
hg qref -d '0 0'
hg qpop
echo universe >> hello.txt
echo universe >> bye.txt

echo % qpush should fail, local changes
hg qpush

echo % apply force, should discard changes in hello, but not bye
hg qpush -f
hg st
hg diff --config diff.nodates=True
hg qdiff --config diff.nodates=True

echo % test popping revisions not in working dir ancestry
hg qseries -v
hg up qparent
hg qpop