largefiles: more safe handling of interruptions while updating modifications stable

largefiles: more safe handling of interruptions while updating modifications

Largefiles are fragile with the design where dirstate and lfdirstate must be
kept in sync.

To be less fragile, mark all clean largefiles as unsure ("normallookup") before
updating standins. After standins have been updated and we know exactly which
largefile standins actually was changed, mark the unchanged largefiles back to
clean ("normal").

This will make the failure mode more safe. If interrupted, the next command
will continue to perform extra hashing of all largefiles. That will do that all
largefiles that are out of sync with their standin will be marked dirty and
they will show up in status and can be cleaned with update --clean.

--- a/hgext/largefiles/overrides.py	Sun Oct 16 02:26:38 2016 +0200
+++ b/hgext/largefiles/overrides.py	Sun Oct 16 02:29:45 2016 +0200
@@ -1390,7 +1390,8 @@
         lfdirstate = lfutil.openlfdirstate(repo.ui, repo)
         unsure, s = lfdirstate.status(matchmod.always(repo.root,
                                                     repo.getcwd()),
-                                      [], False, False, False)
+                                      [], False, True, False)
+        oldclean = set(s.clean)
         pctx = repo['.']
         for lfile in unsure + s.modified:
             lfileabs = repo.wvfs.join(lfile)
@@ -1402,9 +1403,13 @@
                                 lfutil.getexecutable(lfileabs))
             if (standin in pctx and
                 lfhash == lfutil.readstandin(repo, lfile, '.')):
-                lfdirstate.normal(lfile)
+                oldclean.add(lfile)
         for lfile in s.added:
             lfutil.updatestandin(repo, lfutil.standin(lfile))
+        # mark all clean largefiles as dirty, just in case the update gets
+        # interrupted before largefiles and lfdirstate are synchronized
+        for lfile in oldclean:
+            lfdirstate.normallookup(lfile)
         lfdirstate.write()
 
         oldstandins = lfutil.getstandinsstate(repo)
@@ -1413,6 +1418,13 @@
 
         newstandins = lfutil.getstandinsstate(repo)
         filelist = lfutil.getlfilestoupdate(oldstandins, newstandins)
+
+        # to avoid leaving all largefiles as dirty and thus rehash them, mark
+        # all the ones that didn't change as clean
+        for lfile in oldclean.difference(filelist):
+            lfdirstate.normal(lfile)
+        lfdirstate.write()
+
         if branchmerge or force or partial:
             filelist.extend(s.deleted + s.removed)
 

--- a/tests/test-largefiles-update.t	Sun Oct 16 02:26:38 2016 +0200
+++ b/tests/test-largefiles-update.t	Sun Oct 16 02:29:45 2016 +0200
@@ -732,16 +732,16 @@
 
 #endif
 
-Test a fatal error interrupting an update. lfdirstate doesn't realize that
-.hglf has been updated while the largefile hasn't. Status thus shows a clean
-state ... but rebuilding lfdirstate and checking all hashes reveals it isn't
-clean.
+Test a fatal error interrupting an update. Verify that status report dirty
+files correctly after an interrupted update. Also verify that checking all
+hashes reveals it isn't clean.
 
 Start with clean dirstates:
   $ hg up -qcr "8^"
   $ sleep 1
   $ hg st
-Update standins without updating largefiles:
+Update standins without updating largefiles - large1 is modified and largeX is
+added:
   $ cat << EOF > ../crashupdatelfiles.py
   > import hgext.largefiles.lfutil
   > def getlfilestoupdate(oldstandins, newstandins):
@@ -750,29 +750,31 @@
   > EOF
   $ hg up -Cr "8" --config extensions.crashupdatelfiles=../crashupdatelfiles.py
   [7]
-Check large1 content and status:
-BUG: largeX is R and large1 is not M and update does nothing
+Check large1 content and status ... and that update will undo modifications:
+BUG: large is R
   $ cat large1
   large1 in #3
   $ hg st
+  M large1
   R largeX
   $ hg up -Cr .
-  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  getting changed largefiles
+  1 largefiles updated, 0 removed
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ cat large1
+  manually modified before 'hg transplant --continue'
   $ hg st
   R largeX
-Force largefiles rehashing and check again - revealing modifications that
-update now can remove:
+Force largefiles rehashing and check again - which makes it realize that largeX
+not has been removed but just doesn't exist:
   $ rm .hg/largefiles/dirstate
   $ hg st
-  M large1
   ! largeX
   $ hg up -Cr .
   getting changed largefiles
-  2 largefiles updated, 0 removed
-  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  1 largefiles updated, 0 removed
+  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
   $ hg st
-  $ cat large1
-  manually modified before 'hg transplant --continue'
 
   $ cd ..