view tests/test-sidedata.t @ 43709:039fbd14d4e2

lock: fix race in lock-breaking code

With low frequency, I see hg pulls fail with output like:
abort: no such file or directory: .hg/store/lock

I think what happens is, in lock.py, in:

    def _testlock(self, locker):
        if not self._lockshouldbebroken(locker):
            return locker

        # if locker dead, break lock.  must do this with another lock
        # held, or can race and break valid lock.
        try:
            with lock(self.vfs, self.f + b'.break', timeout=0):
                self.vfs.unlink(self.f)
        except error.LockError:
            return locker

if a lock is breakable on disk, and two hg processes concurrently get
to the "if locker dead" comment, a possible interleaving is: process1
finishes executing the function and then process2 finishes executing
the function. If that happens, process2 will either get ENOENT in
self.vfs.unlink (resulting in the spurious failure above), or break a
valid lock and potentially cause repository corruption.

The fix is simple enough: make sure the lock is breakable _inside_ the
critical section, because only then can we know that no other process
can invalidate our knowledge on the lock on disk.

I don't think there are tests for this. I've tested this manually
with:

diff --git a/mercurial/lock.py b/mercurial/lock.py
--- a/mercurial/lock.py
+++ b/mercurial/lock.py
@@ -351,6 +351,8 @@ class lock(object):
         if not self._lockshouldbebroken(locker):
             return locker

+        import random
+        time.sleep(1. + random.random())
         # if locker dead, break lock.  must do this with another lock
         # held, or can race and break valid lock.
         try:
@@ -358,6 +360,7 @@ class lock(object):
                 self.vfs.unlink(self.f)
         except error.LockError:
             return locker
+        time.sleep(1)

     def testlock(self):
         """return id of locker if lock is valid, else None.

and I see this change of behavior before/after this commit:

 $ $hg init repo
 $ cd repo
 $ ln -s $HOSTNAME/effffffc:987654321 .hg/wlock
 $ touch a
 $ $hg commit -Am_ & $hg commit -Am _; wait
-abort: No such file or directory: '/tmp/repo/.hg/wlock'
 adding a
+warning: ignoring unknown working parent 679a8959a8ca!
+nothing changed

Differential Revision: https://phab.mercurial-scm.org/D7199

==========================================================
Test file dedicated to checking side-data related behavior
==========================================================

Check data can be written/read from sidedata
============================================

  $ cat << EOF >> $HGRCPATH
  > [extensions]
  > testsidedata=$TESTDIR/testlib/ext-sidedata.py
  > EOF

  $ hg init test-sidedata --config format.exp-use-side-data=yes
  $ cd test-sidedata
  $ echo aaa > a
  $ hg add a
  $ hg commit -m a --traceback
  $ echo aaa > b
  $ hg add b
  $ hg commit -m b
  $ echo xxx >> a
  $ hg commit -m aa

  $ hg debugsidedata -c 0
  2 sidedata entries
   entry-0001 size 4
   entry-0002 size 32
  $ hg debugsidedata -c 1 -v
  2 sidedata entries
   entry-0001 size 4
    '\x00\x00\x006'
   entry-0002 size 32
    '\x98\t\xf9\xc4v\xf0\xc5P\x90\xf7wRf\xe8\xe27e\xfc\xc1\x93\xa4\x96\xd0\x1d\x97\xaaG\x1d\xd7t\xfa\xde'
  $ hg debugsidedata -m 2
  2 sidedata entries
   entry-0001 size 4
   entry-0002 size 32
  $ hg debugsidedata a  1
  2 sidedata entries
   entry-0001 size 4
   entry-0002 size 32

Check upgrade behavior
======================

Right now, sidedata has not upgrade support

Check that we can upgrade to sidedata
-------------------------------------

  $ hg init up-no-side-data --config format.exp-use-side-data=no
  $ hg debugformat -v -R up-no-side-data
  format-variant    repo config default
  fncache:           yes    yes     yes
  dotencode:         yes    yes     yes
  generaldelta:      yes    yes     yes
  sparserevlog:      yes    yes     yes
  sidedata:           no     no      no
  copies-sdc:         no     no      no
  plain-cl-delta:    yes    yes     yes
  compression:       zlib   zlib    zlib
  compression-level: default default default
  $ hg debugformat -v -R up-no-side-data --config format.exp-use-side-data=yes
  format-variant    repo config default
  fncache:           yes    yes     yes
  dotencode:         yes    yes     yes
  generaldelta:      yes    yes     yes
  sparserevlog:      yes    yes     yes
  sidedata:           no    yes      no
  copies-sdc:         no     no      no
  plain-cl-delta:    yes    yes     yes
  compression:       zlib   zlib    zlib
  compression-level: default default default
  $ hg debugupgraderepo -R up-no-side-data --config format.exp-use-side-data=yes > /dev/null

Check that we can downgrade from sidedata
-----------------------------------------

  $ hg init up-side-data --config format.exp-use-side-data=yes
  $ hg debugformat -v -R up-side-data
  format-variant    repo config default
  fncache:           yes    yes     yes
  dotencode:         yes    yes     yes
  generaldelta:      yes    yes     yes
  sparserevlog:      yes    yes     yes
  sidedata:          yes     no      no
  copies-sdc:         no     no      no
  plain-cl-delta:    yes    yes     yes
  compression:       zlib   zlib    zlib
  compression-level: default default default
  $ hg debugformat -v -R up-side-data --config format.exp-use-side-data=no
  format-variant    repo config default
  fncache:           yes    yes     yes
  dotencode:         yes    yes     yes
  generaldelta:      yes    yes     yes
  sparserevlog:      yes    yes     yes
  sidedata:          yes     no      no
  copies-sdc:         no     no      no
  plain-cl-delta:    yes    yes     yes
  compression:       zlib   zlib    zlib
  compression-level: default default default
  $ hg debugupgraderepo -R up-side-data --config format.exp-use-side-data=no > /dev/null