largefiles: handle merges between normal files and largefiles (issue3084) stable

largefiles: handle merges between normal files and largefiles (issue3084)

The largefiles extension prevents users from adding a normal file
named 'foo' if there is already a largefile with the same name.
However, there was a loop-hole: when merging, it was possible to bring
in a normal file named 'foo' while also having a '.hglf/foo' file.

This patch fixes this by extending the manifest merge to deal with
these kinds of conflicts. If there is a normal file 'foo' in the
working copy, and the other parent brings in a '.hglf/foo' file, then
the user will be prompted to keep the normal file or the largefile.
Likewise for the symmetric case where a normal file is brought in via
the second parent. The prompt looks like this:

  $ hg merge
  foo has been turned into a largefile
  use (l)argefile or keep as (n)ormal file?

After the merge, either the '.hglf/foo' file or the 'foo' file will
have been deleted. This would cause status to return output like:

  $ hg status
  M foo
  R foo

To fix this, the lfiles_repo.status method is changed so that a
removed normal file isn't shown if there is largefile with the same
name, and vice versa for largefiles.

--- a/hgext/largefiles/lfcommands.py	Wed Dec 14 15:41:08 2011 +0100
+++ b/hgext/largefiles/lfcommands.py	Fri Dec 09 17:35:00 2011 +0100
@@ -446,7 +446,11 @@
             os.chmod(abslfile, mode)
             ret = 1
     else:
-        if os.path.exists(abslfile):
+        # Remove lfiles for which the standin is deleted, unless the
+        # lfile is added to the repository again. This happens when a
+        # largefile is converted back to a normal file: the standin
+        # disappears, but a new (normal) file appears as the lfile.
+        if os.path.exists(abslfile) and lfile not in repo[None]:
             os.unlink(abslfile)
             ret = -1
     state = repo.dirstate[lfutil.standin(lfile)]

--- a/hgext/largefiles/overrides.py	Wed Dec 14 15:41:08 2011 +0100
+++ b/hgext/largefiles/overrides.py	Fri Dec 09 17:35:00 2011 +0100
@@ -242,6 +242,90 @@
         wlock.release()
     return orig(ui, repo, *pats, **opts)
 
+# Before starting the manifest merge, merge.updates will call
+# _checkunknown to check if there are any files in the merged-in
+# changeset that collide with unknown files in the working copy.
+#
+# The largefiles are seen as unknown, so this prevents us from merging
+# in a file 'foo' if we already have a largefile with the same name.
+#
+# The overridden function filters the unknown files by removing any
+# largefiles. This makes the merge proceed and we can then handle this
+# case further in the overridden manifestmerge function below.
+def override_checkunknown(origfn, wctx, mctx, folding):
+    origunknown = wctx.unknown()
+    wctx._unknown = filter(lambda f: lfutil.standin(f) not in wctx, origunknown)
+    try:
+        return origfn(wctx, mctx, folding)
+    finally:
+        wctx._unknown = origunknown
+
+# The manifest merge handles conflicts on the manifest level. We want
+# to handle changes in largefile-ness of files at this level too.
+#
+# The strategy is to run the original manifestmerge and then process
+# the action list it outputs. There are two cases we need to deal with:
+#
+# 1. Normal file in p1, largefile in p2. Here the largefile is
+#    detected via its standin file, which will enter the working copy
+#    with a "get" action. It is not "merge" since the standin is all
+#    Mercurial is concerned with at this level -- the link to the
+#    existing normal file is not relevant here.
+#
+# 2. Largefile in p1, normal file in p2. Here we get a "merge" action
+#    since the largefile will be present in the working copy and
+#    different from the normal file in p2. Mercurial therefore
+#    triggers a merge action.
+#
+# In both cases, we prompt the user and emit new actions to either
+# remove the standin (if the normal file was kept) or to remove the
+# normal file and get the standin (if the largefile was kept). The
+# default prompt answer is to use the largefile version since it was
+# presumably changed on purpose.
+#
+# Finally, the merge.applyupdates function will then take care of
+# writing the files into the working copy and lfcommands.updatelfiles
+# will update the largefiles.
+def override_manifestmerge(origfn, repo, p1, p2, pa, overwrite, partial):
+    actions = origfn(repo, p1, p2, pa, overwrite, partial)
+    processed = []
+
+    for action in actions:
+        if overwrite:
+            processed.append(action)
+            continue
+        f, m = action[:2]
+
+        choices = (_('&Largefile'), _('&Normal file'))
+        if m == "g" and lfutil.splitstandin(f) in p1 and f in p2:
+            # Case 1: normal file in the working copy, largefile in
+            # the second parent
+            lfile = lfutil.splitstandin(f)
+            standin = f
+            msg = _('%s has been turned into a largefile\n'
+                    'use (l)argefile or keep as (n)ormal file?') % lfile
+            if repo.ui.promptchoice(msg, choices, 0) == 0:
+                processed.append((lfile, "r"))
+                processed.append((standin, "g", p2.flags(standin)))
+            else:
+                processed.append((standin, "r"))
+        elif m == "m" and lfutil.standin(f) in p1 and f in p2:
+            # Case 2: largefile in the working copy, normal file in
+            # the second parent
+            standin = lfutil.standin(f)
+            lfile = f
+            msg = _('%s has been turned into a normal file\n'
+                    'keep as (l)argefile or use (n)ormal file?') % lfile
+            if repo.ui.promptchoice(msg, choices, 0) == 0:
+                processed.append((lfile, "r"))
+            else:
+                processed.append((standin, "r"))
+                processed.append((lfile, "g", p2.flags(lfile)))
+        else:
+            processed.append(action)
+
+    return processed
+
 # Override filemerge to prompt the user about how they wish to merge
 # largefiles. This will handle identical edits, and copy/rename +
 # edit without prompting the user.

--- a/hgext/largefiles/reposetup.py	Wed Dec 14 15:41:08 2011 +0100
+++ b/hgext/largefiles/reposetup.py	Fri Dec 09 17:35:00 2011 +0100
@@ -192,9 +192,18 @@
                             continue
                         if lfile not in lfdirstate:
                             removed.append(lfile)
-                    # Handle unknown and ignored differently
-                    lfiles = (modified, added, removed, missing, [], [], clean)
+
+                    # Filter result lists
                     result = list(result)
+
+                    # Largefiles are not really removed when they're
+                    # still in the normal dirstate. Likewise, normal
+                    # files are not really removed if it's still in
+                    # lfdirstate. This happens in merges where files
+                    # change type.
+                    removed = [f for f in removed if f not in repo.dirstate]
+                    result[2] = [f for f in result[2] if f not in lfdirstate]
+
                     # Unknown files
                     result[4] = [f for f in unknown
                                  if (repo.dirstate[f] == '?' and
@@ -206,6 +215,7 @@
                     normals = [[fn for fn in filelist
                                 if not lfutil.isstandin(fn)]
                                for filelist in result]
+                    lfiles = (modified, added, removed, missing, [], [], clean)
                     result = [sorted(list1 + list2)
                               for (list1, list2) in zip(normals, lfiles)]
                 else:

--- a/hgext/largefiles/uisetup.py	Wed Dec 14 15:41:08 2011 +0100
+++ b/hgext/largefiles/uisetup.py	Fri Dec 09 17:35:00 2011 +0100
@@ -9,7 +9,7 @@
 '''setup for largefiles extension: uisetup'''
 
 from mercurial import archival, cmdutil, commands, extensions, filemerge, hg, \
-    httprepo, localrepo, sshrepo, sshserver, wireproto
+    httprepo, localrepo, merge, sshrepo, sshserver, wireproto
 from mercurial.i18n import _
 from mercurial.hgweb import hgweb_mod, protocol
 
@@ -62,6 +62,10 @@
                                    overrides.override_update)
     entry = extensions.wrapcommand(commands.table, 'pull',
                                    overrides.override_pull)
+    entry = extensions.wrapfunction(merge, '_checkunknown',
+                                    overrides.override_checkunknown)
+    entry = extensions.wrapfunction(merge, 'manifestmerge',
+                                    overrides.override_manifestmerge)
     entry = extensions.wrapfunction(filemerge, 'filemerge',
                                     overrides.override_filemerge)
     entry = extensions.wrapfunction(cmdutil, 'copy',

--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/tests/test-issue3084.t	Fri Dec 09 17:35:00 2011 +0100
@@ -0,0 +1,108 @@
+
+  $ echo "[extensions]" >> $HGRCPATH
+  $ echo "largefiles =" >> $HGRCPATH
+
+Create the repository outside $HOME since largefiles write to
+$HOME/.cache/largefiles.
+
+  $ hg init test
+  $ cd test
+  $ echo "root" > root
+  $ hg add root
+  $ hg commit -m "Root commit"
+
+  $ echo "large" > foo
+  $ hg add --large foo
+  $ hg commit -m "Add foo as a largefile"
+
+  $ hg update -r 0
+  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  getting changed largefiles
+  0 largefiles updated, 1 removed
+
+  $ echo "normal" > foo
+  $ hg add foo
+  $ hg commit -m "Add foo as normal file"
+  created new head
+
+Normal file in the working copy, keeping the normal version:
+
+  $ echo "n" | hg merge --config ui.interactive=Yes
+  foo has been turned into a largefile
+  use (l)argefile or keep as (n)ormal file? 0 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  (branch merge, don't forget to commit)
+
+  $ hg status
+  $ cat foo
+  normal
+
+Normal file in the working copy, keeping the largefile version:
+
+  $ hg update -q -C
+  $ echo "l" | hg merge --config ui.interactive=Yes
+  foo has been turned into a largefile
+  use (l)argefile or keep as (n)ormal file? 1 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  (branch merge, don't forget to commit)
+  getting changed largefiles
+  1 largefiles updated, 0 removed
+
+  $ hg status
+  M foo
+
+  $ hg diff --nodates
+  diff -r fa129ab6b5a7 .hglf/foo
+  --- /dev/null
+  +++ b/.hglf/foo
+  @@ -0,0 +1,1 @@
+  +7f7097b041ccf68cc5561e9600da4655d21c6d18
+  diff -r fa129ab6b5a7 foo
+  --- a/foo
+  +++ /dev/null
+  @@ -1,1 +0,0 @@
+  -normal
+
+  $ cat foo
+  large
+
+Largefile in the working copy, keeping the normal version:
+
+  $ hg update -q -C -r 1
+  $ echo "n" | hg merge --config ui.interactive=Yes
+  foo has been turned into a normal file
+  keep as (l)argefile or use (n)ormal file? 1 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  (branch merge, don't forget to commit)
+  getting changed largefiles
+  0 largefiles updated, 0 removed
+
+  $ hg status
+  M foo
+
+  $ hg diff --nodates
+  diff -r ff521236428a .hglf/foo
+  --- a/.hglf/foo
+  +++ /dev/null
+  @@ -1,1 +0,0 @@
+  -7f7097b041ccf68cc5561e9600da4655d21c6d18
+  diff -r ff521236428a foo
+  --- /dev/null
+  +++ b/foo
+  @@ -0,0 +1,1 @@
+  +normal
+
+  $ cat foo
+  normal
+
+Largefile in the working copy, keeping the largefile version:
+
+  $ hg update -q -C -r 1
+  $ echo "l" | hg merge --config ui.interactive=Yes
+  foo has been turned into a normal file
+  keep as (l)argefile or use (n)ormal file? 0 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  (branch merge, don't forget to commit)
+  getting changed largefiles
+  1 largefiles updated, 0 removed
+
+  $ hg status
+
+  $ cat foo
+  large