named branches: server branchmap wire protocol support (issue736)
The repository command, 'branchmap', returns a dictionary, branchname
-> [branchheads], and will be implemented for localrepo, httprepo and
sshrepo.
The following wire format is used for returning data:
branchname1 branch1head2 branch1head2 ...
branchname2 ...
...
Branch names are URL encoded to escape white space, and branch heads
are sent as hex encoded node ids. All branches and all their heads are
sent.
The background and motivation for this command is the desire for a
richer named branch semantics when pushing changesets. The details are
explained in the original proposal which is included below.
1. BACKGROUND
The algorithm currently implemented in Mercurial only considers the
graph theoretical heads when determining whether new heads are
created, rather than using the branch heads as a count (the algorithm
considers a branch head effectively closed when it is merged into
another branch or a new named branch is started from that point
onward).
Our particular problem with the algorithm is that we'd like to see the
following case working without forcing a push:
Upsteam has:
(0:dev) ---- (1:dev)
\
`--- (2:stable)
Someone merges stable into dev:
(0:dev) ---- (1:dev) ------(3:dev)
\ /
`--- (2:stable) --------´
This can be pushed without --force (as it should).
Now someone else does some coding on stable (a bug fix, say):
(0:dev) ---- (1:dev) ------(3:dev)
\ /
`--- (2:stable) ---------´---------(4:stable)
This time we need --force to push.
We allow this to be pushed without using --force by getting all the
remote branch heads (by extending the wire protocol with a new
function).
We would, furthermore, also prefer if it is impossible to push a new
branch without --force (or a later --newbranch option so --force isn't
shoe-horned into too many disparate functions, if need be), except of
course in the case where the remote repository is empty.
This is what our patches accomplish.
2. ALTERNATIVES
We have, of course, considered some alternatives to reconstructing
enough information to decide whether we are creating new remote branch
heads, before we added the new wire protocol command.
2.1. LOOKUP ON REMOTE
The main alternative is to use the information from remote.heads() and
remote.lookup() to try to reconstruct enough graph information to
decide whether we are creating new heads. This is not adequate as
illustrated below.
Remember that each lookup is typically a request-response pair over
SSH or HTTP(S).
If we have a simple repository at the remote end like this:
(0:dev) ---- (1:dev) ---- (3:stable)
\
`--- (2:dev)
then remote.heads() will yield [2, 3]. Assume we have nodes [0, 1, 2]
locally and want to create a new node, 4:dev, as a descendant from
(1:dev), which should be OK as 1:dev is a branch head.
If we do remote.lookup('dev') we will get [2]. Thus, we can get
information about whether a branch exists on the remote server or not,
but this does not solve our problem of figuring out whether we are
creating new heads or not.
Pushing 4:dev ought to be OK, since after the push, we still only have
two heads on branch a.
Using remote.lookup() and remote.heads() is thus not adequate to
consistently decide whether we are creating new remote heads (e.g. in
this situation the latter would never return 1:dev).
2.2. USING INCOMING TO RECONSTRUCT THE GRAPH
An alternative would be to use information equivalent to hg incoming
to get the full remote graph in addition to the local graph.
To do this, we would have to get a changegroup(subset) bundle
representing the remote end (which may be a substantial amount of
data), getting the branch heads from an instantiated bundlerepository,
deleting the bundle, and finally, we can compute the prepush logic.
While this is backwards compatible, it will cause a possibly
substantial slowdown of the push command as it first needs to pull in
all changes.
3. FURTHER ARGUMENTS IN FAVOUR OF THE BRANCHMAP WIRE-PROTOCOL EXTENSION
Currently, the commands incoming and pull, work based on the tip of a
given branch if used with "-r branchname", making it hard to get all
revisions of a certain branch only (if it has multiple heads). This
can be solved by requesting the remote's branchheads and letting the
revisions to be used with the command be these heads. This can be done
by extending the commands with a new option, e.g.:
hg pull -b branchname
which will be turned into the equivalent of:
hg pull -r branchhead1 -r branchhead2 -r branchhead3
We have a simple follow-up patch that can do this ready as well
(although not submitted yet as it is pending the acceptance of the
branch patch).
4. WRAP-UP
We generally find that the branchmap wire protocol extension can
provide better named branch support to Mercurial. Currently, some
things, like the initial push scenario in this mail, are fairly
counter-intuitive, and the more often you have to force push, the more
it is likely you will get a lot of spurious and unnecessary merge
nodes. Also, restricting incoming and pull to all changes on a branch
rather than changes on the tip-most head would be a sensible extension
to making named branches a first class citizen in Mercurial.
Currently, named branches sometimes feel like a late-coming unwanted
step-child.
We have run it in a production environment for a while, with fewer
multiple heads occurring in our repositories and fewer confused users
as a result.
Also, it fixes the long-standing issue 736.
Co-contributor: Sune Foldager <cryo@cyanite.org>
hg convert [OPTION]... SOURCE [DEST [REVMAP]]
convert a foreign SCM repository to a Mercurial one.
Accepted source formats [identifiers]:
- Mercurial [hg]
- CVS [cvs]
- Darcs [darcs]
- git [git]
- Subversion [svn]
- Monotone [mtn]
- GNU Arch [gnuarch]
- Bazaar [bzr]
- Perforce [p4]
Accepted destination formats [identifiers]:
- Mercurial [hg]
- Subversion [svn] (history on branches is not preserved)
If no revision is given, all revisions will be converted.
Otherwise, convert will only import up to the named revision
(given in a format understood by the source).
If no destination directory name is specified, it defaults to the
basename of the source with '-hg' appended. If the destination
repository doesn't exist, it will be created.
If <REVMAP> isn't given, it will be put in a default location
(<dest>/.hg/shamap by default). The <REVMAP> is a simple text file
that maps each source commit ID to the destination ID for that
revision, like so:
<source ID> <destination ID>
If the file doesn't exist, it's automatically created. It's
updated on each commit copied, so convert-repo can be interrupted
and can be run repeatedly to copy new commits.
The [username mapping] file is a simple text file that maps each
source commit author to a destination commit author. It is handy
for source SCMs that use unix logins to identify authors (eg:
CVS). One line per author mapping and the line format is:
srcauthor=whatever string you want
The filemap is a file that allows filtering and remapping of files
and directories. Comment lines start with '#'. Each line can
contain one of the following directives:
include path/to/file
exclude path/to/file
rename from/file to/file
The 'include' directive causes a file, or all files under a
directory, to be included in the destination repository, and the
exclusion of all other files and directories not explicitely included.
The 'exclude' directive causes files or directories to be omitted.
The 'rename' directive renames a file or directory. To rename from
a subdirectory into the root of the repository, use '.' as the
path to rename to.
The splicemap is a file that allows insertion of synthetic
history, letting you specify the parents of a revision. This is
useful if you want to e.g. give a Subversion merge two parents, or
graft two disconnected series of history together. Each entry
contains a key, followed by a space, followed by one or two
comma-separated values. The key is the revision ID in the source
revision control system whose parents should be modified (same
format as a key in .hg/shamap). The values are the revision IDs
(in either the source or destination revision control system) that
should be used as the new parents for that node.
The branchmap is a file that allows you to rename a branch when it is
being brought in from whatever external repository. When used in
conjunction with a splicemap, it allows for a powerful combination
to help fix even the most badly mismanaged repositories and turn them
into nicely structured Mercurial repositories. The branchmap contains
lines of the form "original_branch_name new_branch_name".
"original_branch_name" is the name of the branch in the source
repository, and "new_branch_name" is the name of the branch is the
destination repository. This can be used to (for instance) move code
in one repository from "default" to a named branch.
Mercurial Source
-----------------
--config convert.hg.ignoreerrors=False (boolean)
ignore integrity errors when reading. Use it to fix Mercurial
repositories with missing revlogs, by converting from and to
Mercurial.
--config convert.hg.saverev=False (boolean)
store original revision ID in changeset (forces target IDs to
change)
--config convert.hg.startrev=0 (hg revision identifier)
convert start revision and its descendants
CVS Source
----------
CVS source will use a sandbox (i.e. a checked-out copy) from CVS
to indicate the starting point of what will be converted. Direct
access to the repository files is not needed, unless of course the
repository is :local:. The conversion uses the top level directory
in the sandbox to find the CVS repository, and then uses CVS rlog
commands to find files to convert. This means that unless a
filemap is given, all files under the starting directory will be
converted, and that any directory reorganisation in the CVS
sandbox is ignored.
Because CVS does not have changesets, it is necessary to collect
individual commits to CVS and merge them into changesets. CVS
source uses its internal changeset merging code by default but can
be configured to call the external 'cvsps' program by setting:
--config convert.cvsps='cvsps -A -u --cvs-direct -q'
This is a legacy option and may be removed in future.
The options shown are the defaults.
Internal cvsps is selected by setting
--config convert.cvsps=builtin
and has a few more configurable options:
--config convert.cvsps.cache=True (boolean)
Set to False to disable remote log caching, for testing and
debugging purposes.
--config convert.cvsps.fuzz=60 (integer)
Specify the maximum time (in seconds) that is allowed
between commits with identical user and log message in a
single changeset. When very large files were checked in as
part of a changeset then the default may not be long
enough.
--config convert.cvsps.mergeto='{{mergetobranch ([-\w]+)}}'
Specify a regular expression to which commit log messages
are matched. If a match occurs, then the conversion
process will insert a dummy revision merging the branch on
which this log message occurs to the branch indicated in
the regex.
--config convert.cvsps.mergefrom='{{mergefrombranch ([-\w]+)}}'
Specify a regular expression to which commit log messages
are matched. If a match occurs, then the conversion
process will add the most recent revision on the branch
indicated in the regex as the second parent of the
changeset.
The hgext/convert/cvsps wrapper script allows the builtin
changeset merging code to be run without doing a conversion. Its
parameters and output are similar to that of cvsps 2.1.
Subversion Source
-----------------
Subversion source detects classical trunk/branches/tags layouts.
By default, the supplied "svn://repo/path/" source URL is
converted as a single branch. If "svn://repo/path/trunk" exists it
replaces the default branch. If "svn://repo/path/branches" exists,
its subdirectories are listed as possible branches. If
"svn://repo/path/tags" exists, it is looked for tags referencing
converted branches. Default "trunk", "branches" and "tags" values
can be overriden with following options. Set them to paths
relative to the source URL, or leave them blank to disable
autodetection.
--config convert.svn.branches=branches (directory name)
specify the directory containing branches
--config convert.svn.tags=tags (directory name)
specify the directory containing tags
--config convert.svn.trunk=trunk (directory name)
specify the name of the trunk branch
Source history can be retrieved starting at a specific revision,
instead of being integrally converted. Only single branch
conversions are supported.
--config convert.svn.startrev=0 (svn revision number)
specify start Subversion revision.
Perforce Source
---------------
The Perforce (P4) importer can be given a p4 depot path or a
client specification as source. It will convert all files in the
source to a flat Mercurial repository, ignoring labels, branches
and integrations. Note that when a depot path is given you then
usually should specify a target directory, because otherwise the
target may be named ...-hg.
It is possible to limit the amount of source history to be
converted by specifying an initial Perforce revision.
--config convert.p4.startrev=0 (perforce changelist number)
specify initial Perforce revision.
Mercurial Destination
---------------------
--config convert.hg.clonebranches=False (boolean)
dispatch source branches in separate clones.
--config convert.hg.tagsbranch=default (branch name)
tag revisions branch name
--config convert.hg.usebranchnames=True (boolean)
preserve branch names
options:
-A --authors username mapping filename
-d --dest-type destination repository type
--filemap remap file names using contents of file
-r --rev import up to target revision REV
-s --source-type source repository type
--splicemap splice synthesized history into place
--branchmap change branch names while converting
--datesort try to sort changesets by date
use "hg -v help convert" to show global options
adding a
assuming destination a-hg
initializing destination a-hg repository
scanning source...
sorting...
converting...
4 a
3 b
2 c
1 d
0 e
pulling from ../a
searching for changes
no changes found
% should fail
initializing destination bogusfile repository
abort: cannot create new bundle repository
% should fail
abort: Permission denied: bogusdir
% should succeed
initializing destination bogusdir repository
scanning source...
sorting...
converting...
4 a
3 b
2 c
1 d
0 e
% test pre and post conversion actions
run hg source pre-conversion action
run hg sink pre-conversion action
run hg sink post-conversion action
run hg source post-conversion action
% converting empty dir should fail nicely
assuming destination emptydir-hg
initializing destination emptydir-hg repository
emptydir does not look like a CVS checkout
emptydir does not look like a Git repo
emptydir does not look like a Subversion repo
emptydir is not a local Mercurial repo
emptydir does not look like a darcs repo
emptydir does not look like a monotone repo
emptydir does not look like a GNU Arch repo
emptydir does not look like a Bazaar repo
emptydir does not look like a P4 repo
abort: emptydir: missing or unsupported repository