tests/test-import

named branches: server branchmap wire protocol support (issue736)

The repository command, 'branchmap', returns a dictionary, branchname
-> [branchheads], and will be implemented for localrepo, httprepo and
sshrepo.

The following wire format is used for returning data:

branchname1 branch1head2 branch1head2 ...
branchname2 ...
...

Branch names are URL encoded to escape white space, and branch heads
are sent as hex encoded node ids. All branches and all their heads are
sent.

The background and motivation for this command is the desire for a
richer named branch semantics when pushing changesets. The details are
explained in the original proposal which is included below.


1. BACKGROUND

The algorithm currently implemented in Mercurial only considers the
graph theoretical heads when determining whether new heads are
created, rather than using the branch heads as a count (the algorithm
considers a branch head effectively closed when it is merged into
another branch or a new named branch is started from that point
onward).

Our particular problem with the algorithm is that we'd like to see the
following case working without forcing a push:

Upsteam has:

(0:dev) ---- (1:dev)
\
 `--- (2:stable)

Someone merges stable into dev:

(0:dev) ---- (1:dev) ------(3:dev)
\                         /
 `--- (2:stable) --------´

This can be pushed without --force (as it should).
Now someone else does some coding on stable (a bug fix, say):

(0:dev) ---- (1:dev) ------(3:dev)
\                          /
 `--- (2:stable) ---------´---------(4:stable)

This time we need --force to push.

We allow this to be pushed without using --force by getting all the
remote branch heads (by extending the wire protocol with a new
function).

We would, furthermore, also prefer if it is impossible to push a new
branch without --force (or a later --newbranch option so --force isn't
shoe-horned into too many disparate functions, if need be), except of
course in the case where the remote repository is empty.

This is what our patches accomplish.


2. ALTERNATIVES

We have, of course, considered some alternatives to reconstructing
enough information to decide whether we are creating new remote branch
heads, before we added the new wire protocol command.

2.1. LOOKUP ON REMOTE

The main alternative is to use the information from remote.heads() and
remote.lookup() to try to reconstruct enough graph information to
decide whether we are creating new heads. This is not adequate as
illustrated below.

Remember that each lookup is typically a request-response pair over
SSH or HTTP(S).

If we have a simple repository at the remote end like this:

(0:dev) ---- (1:dev) ---- (3:stable)
\
 `--- (2:dev)

then remote.heads() will yield [2, 3]. Assume we have nodes [0, 1, 2]
locally and want to create a new node, 4:dev, as a descendant from
(1:dev), which should be OK as 1:dev is a branch head.

If we do remote.lookup('dev') we will get [2]. Thus, we can get
information about whether a branch exists on the remote server or not,
but this does not solve our problem of figuring out whether we are
creating new heads or not.

Pushing 4:dev ought to be OK, since after the push, we still only have
two heads on branch a.

Using remote.lookup() and remote.heads() is thus not adequate to
consistently decide whether we are creating new remote heads (e.g. in
this situation the latter would never return 1:dev).

2.2. USING INCOMING TO RECONSTRUCT THE GRAPH

An alternative would be to use information equivalent to hg incoming
to get the full remote graph in addition to the local graph.

To do this, we would have to get a changegroup(subset) bundle
representing the remote end (which may be a substantial amount of
data), getting the branch heads from an instantiated bundlerepository,
deleting the bundle, and finally, we can compute the prepush logic.

While this is backwards compatible, it will cause a possibly
substantial slowdown of the push command as it first needs to pull in
all changes.


3. FURTHER ARGUMENTS IN FAVOUR OF THE BRANCHMAP WIRE-PROTOCOL EXTENSION

Currently, the commands incoming and pull, work based on the tip of a
given branch if used with "-r branchname", making it hard to get all
revisions of a certain branch only (if it has multiple heads). This
can be solved by requesting the remote's branchheads and letting the
revisions to be used with the command be these heads. This can be done
by extending the commands with a new option, e.g.:

hg pull -b branchname

which will be turned into the equivalent of:

hg pull -r branchhead1 -r branchhead2 -r branchhead3

We have a simple follow-up patch that can do this ready as well
(although not submitted yet as it is pending the acceptance of the
branch patch).


4. WRAP-UP

We generally find that the branchmap wire protocol extension can
provide better named branch support to Mercurial. Currently, some
things, like the initial push scenario in this mail, are fairly
counter-intuitive, and the more often you have to force push, the more
it is likely you will get a lot of spurious and unnecessary merge
nodes. Also, restricting incoming and pull to all changes on a branch
rather than changes on the tip-most head would be a sensible extension
to making named branches a first class citizen in Mercurial.
Currently, named branches sometimes feel like a late-coming unwanted
step-child.

We have run it in a production environment for a while, with fewer
multiple heads occurring in our repositories and fewer confused users
as a result.

Also, it fixes the long-standing issue 736.

Co-contributor: Sune Foldager <[e-mail]>

#!/bin/sh

hg init a
mkdir a/d1
mkdir a/d1/d2
echo line 1 > a/a
echo line 1 > a/d1/d2/a
hg --cwd a ci -Ama

echo line 2 >> a/a
hg --cwd a ci -u someone -d '1 0' -m'second change'

echo % import exported patch
hg clone -r0 a b
hg --cwd a export tip > tip.patch
hg --cwd b import ../tip.patch
echo % message should be same
hg --cwd b tip | grep 'second change'
echo % committer should be same
hg --cwd b tip | grep someone
rm -r b

echo % import exported patch with external patcher
cat > dummypatch.py <<EOF
print 'patching file a'
file('a', 'wb').write('line2\n')
EOF
chmod +x dummypatch.py
hg clone -r0 a b
hg --cwd a export tip > tip.patch
hg --config ui.patch='python ../dummypatch.py' --cwd b import ../tip.patch
cat b/a
rm -r b

echo % import of plain diff should fail without message
hg clone -r0 a b
hg --cwd a diff -r0:1 > tip.patch
hg --cwd b import ../tip.patch
rm -r b

echo % import of plain diff should be ok with message
hg clone -r0 a b
hg --cwd a diff -r0:1 > tip.patch
hg --cwd b import -mpatch ../tip.patch
rm -r b

echo % import of plain diff with specific date and user
hg clone -r0 a b
hg --cwd a diff -r0:1 > tip.patch
hg --cwd b import -mpatch -d '1 0' -u '[e-mail]' ../tip.patch
hg -R b tip -pv
rm -r b

echo % import of plain diff should be ok with --no-commit
hg clone -r0 a b
hg --cwd a diff -r0:1 > tip.patch
hg --cwd b import --no-commit ../tip.patch
hg --cwd b diff --nodates
rm -r b

echo % hg -R repo import
# put the clone in a subdir - having a directory named "a"
# used to hide a bug.
mkdir dir
hg clone -r0 a dir/b
hg --cwd a export tip > dir/tip.patch
cd dir
hg -R b import tip.patch
cd ..
rm -r dir

echo % import from stdin
hg clone -r0 a b
hg --cwd a export tip | hg --cwd b import -
rm -r b

echo % override commit message
hg clone -r0 a b
hg --cwd a export tip | hg --cwd b import -m 'override' -
hg --cwd b tip | grep override
rm -r b

cat > mkmsg.py <<EOF
import email.Message, sys
msg = email.Message.Message()
msg.set_payload('email commit message\n' + open('tip.patch', 'rb').read())
msg['Subject'] = 'email patch'
msg['From'] = 'email patcher'
sys.stdout.write(msg.as_string())
EOF

echo % plain diff in email, subject, message body
hg clone -r0 a b
hg --cwd a diff -r0:1 > tip.patch
python mkmsg.py > msg.patch
hg --cwd b import ../msg.patch
hg --cwd b tip | grep email
rm -r b

echo % plain diff in email, no subject, message body
hg clone -r0 a b
grep -v '^Subject:' msg.patch | hg --cwd b import -
rm -r b

echo % plain diff in email, subject, no message body
hg clone -r0 a b
grep -v '^email ' msg.patch | hg --cwd b import -
rm -r b

echo % plain diff in email, no subject, no message body, should fail
hg clone -r0 a b
egrep -v '^(Subject|email)' msg.patch | hg --cwd b import -
rm -r b

echo % hg export in email, should use patch header
hg clone -r0 a b
hg --cwd a export tip > tip.patch
python mkmsg.py | hg --cwd b import -
hg --cwd b tip | grep second
rm -r b

# subject: duplicate detection, removal of [PATCH]
# The '---' tests the gitsendmail handling without proper mail headers
cat > mkmsg2.py <<EOF
import email.Message, sys
msg = email.Message.Message()
msg.set_payload('email patch\n\nnext line\n---\n' + open('tip.patch').read())
msg['Subject'] = '[PATCH] email patch'
msg['From'] = 'email patcher'
sys.stdout.write(msg.as_string())
EOF

echo '% plain diff in email, [PATCH] subject, message body with subject'
hg clone -r0 a b
hg --cwd a diff -r0:1 > tip.patch
python mkmsg2.py | hg --cwd b import -
hg --cwd b tip --template '{desc}\n'
rm -r b

# We weren't backing up the correct dirstate file when importing many patches
# (issue963)
echo '% import patch1 patch2; rollback'
echo line 3 >> a/a
hg --cwd a ci -m'third change'
hg --cwd a export -o '../patch%R' 1 2
hg clone -qr0 a b
hg --cwd b parents --template 'parent: {rev}\n'
hg --cwd b import ../patch1 ../patch2
hg --cwd b rollback
hg --cwd b parents --template 'parent: {rev}\n'
rm -r b

# bug non regression test
# importing a patch in a subdirectory failed at the commit stage
echo line 2 >> a/d1/d2/a
hg --cwd a ci -u someoneelse -d '1 0' -m'subdir change'
echo % hg import in a subdirectory
hg clone -r0 a b
hg --cwd a export tip | sed -e 's/d1\/d2\///' > tip.patch
dir=`pwd`
cd b/d1/d2 2>&1 > /dev/null
hg import  ../../../tip.patch
cd $dir
echo "% message should be 'subdir change'"
hg --cwd b tip | grep 'subdir change'
echo "% committer should be 'someoneelse'"
hg --cwd b tip | grep someoneelse
echo "% should be empty"
hg --cwd b status


# Test fuzziness (ambiguous patch location, fuzz=2)
echo % test fuzziness
hg init fuzzy
cd fuzzy
echo line1 > a
echo line0 >> a
echo line3 >> a
hg ci -Am adda
echo line1 > a
echo line2 >> a
echo line0 >> a
echo line3 >> a
hg ci -m change a
hg export tip > tip.patch
hg up -C 0
echo line1 > a
echo line0 >> a
echo line1 >> a
echo line0 >> a
hg ci -m brancha
hg import -v tip.patch
cd ..

# Test hunk touching empty files (issue906)
hg init empty
cd empty
touch a
touch b1
touch c1
echo d > d
hg ci -Am init
echo a > a
echo b > b1
hg mv b1 b2
echo c > c1
hg copy c1 c2
rm d
touch d
hg diff --git
hg ci -m empty
hg export --git tip > empty.diff
hg up -C 0
hg import empty.diff
for name in a b1 b2 c1 c2 d;
do
    echo % $name file
    test -f $name && cat $name
done
cd ..

# Test importing a patch ending with a binary file removal
echo % test trailing binary removal
hg init binaryremoval
cd binaryremoval
echo a > a
python -c "file('b', 'wb').write('a\x00b')"
hg ci -Am addall
hg rm a
hg rm b
hg st
hg ci -m remove
hg export --git . > remove.diff
cat remove.diff | grep git
hg up -C 0
hg import remove.diff
hg manifest
cd ..

echo % 'test update+rename with common name (issue 927)'
hg init t
cd t
touch a
hg ci -Am t
echo a > a
# Here, bfile.startswith(afile)
hg copy a a2
hg ci -m copya
hg export --git tip > copy.diff
hg up -C 0
hg import copy.diff
echo % view a
# a should contain an 'a'
cat a
echo % view a2
# and a2 should have duplicated it
cat a2
cd ..

echo % 'test -p0'
hg init p0
cd p0
echo a > a
hg ci -Am t
hg import -p0 - << EOF
foobar
--- a	Sat Apr 12 22:43:58 2008 -0400
+++ a	Sat Apr 12 22:44:05 2008 -0400
@@ -1,1 +1,1 @@
-a
+bb
EOF
hg status
cat a
cd ..

echo % 'test paths outside repo root'
mkdir outside
touch outside/foo
hg init inside
cd inside
hg import - <<EOF
diff --git a/a b/b
rename from ../outside/foo
rename to bar
EOF
cd ..

echo '% test import with similarity (issue295)'
hg init sim
cd sim
echo 'this is a test' > a
hg ci -Ama
cat > ../rename.diff <<EOF
diff --git a/a b/a
deleted file mode 100644
--- a/a
+++ /dev/null
@@ -1,1 +0,0 @@
-this is a test
diff --git a/b b/b
new file mode 100644
--- /dev/null
+++ b/b
@@ -0,0 +1,2 @@
+this is a test
+foo
EOF
hg import --no-commit -v -s 1 ../rename.diff
hg st -C
hg revert -a
rm b
hg import --no-commit -v -s 100 ../rename.diff
hg st -C
cd ..


echo '% add empty file from the end of patch (issue 1495)'
hg init addemptyend
cd addemptyend
touch a
hg addremove
hg ci -m "commit"
cat > a.patch <<EOF
diff --git a/a b/a
--- a/a
+++ b/a
@@ -0,0 +1,1 @@
+a
diff --git a/b b/b
new file mode 100644
EOF
hg import --no-commit a.patch
cd ..