view tests/test-ui-verbosity.py.out @ 43709:039fbd14d4e2

lock: fix race in lock-breaking code

With low frequency, I see hg pulls fail with output like:
abort: no such file or directory: .hg/store/lock

I think what happens is, in lock.py, in:

    def _testlock(self, locker):
        if not self._lockshouldbebroken(locker):
            return locker

        # if locker dead, break lock.  must do this with another lock
        # held, or can race and break valid lock.
        try:
            with lock(self.vfs, self.f + b'.break', timeout=0):
                self.vfs.unlink(self.f)
        except error.LockError:
            return locker

if a lock is breakable on disk, and two hg processes concurrently get
to the "if locker dead" comment, a possible interleaving is: process1
finishes executing the function and then process2 finishes executing
the function. If that happens, process2 will either get ENOENT in
self.vfs.unlink (resulting in the spurious failure above), or break a
valid lock and potentially cause repository corruption.

The fix is simple enough: make sure the lock is breakable _inside_ the
critical section, because only then can we know that no other process
can invalidate our knowledge on the lock on disk.

I don't think there are tests for this. I've tested this manually
with:

diff --git a/mercurial/lock.py b/mercurial/lock.py
--- a/mercurial/lock.py
+++ b/mercurial/lock.py
@@ -351,6 +351,8 @@ class lock(object):
         if not self._lockshouldbebroken(locker):
             return locker

+        import random
+        time.sleep(1. + random.random())
         # if locker dead, break lock.  must do this with another lock
         # held, or can race and break valid lock.
         try:
@@ -358,6 +360,7 @@ class lock(object):
                 self.vfs.unlink(self.f)
         except error.LockError:
             return locker
+        time.sleep(1)

     def testlock(self):
         """return id of locker if lock is valid, else None.

and I see this change of behavior before/after this commit:

 $ $hg init repo
 $ cd repo
 $ ln -s $HOSTNAME/effffffc:987654321 .hg/wlock
 $ touch a
 $ $hg commit -Am_ & $hg commit -Am _; wait
-abort: No such file or directory: '/tmp/repo/.hg/wlock'
 adding a
+warning: ignoring unknown working parent 679a8959a8ca!
+nothing changed

Differential Revision: https://phab.mercurial-scm.org/D7199

      hgrc settings    command line options      final result   
    quiet verbo debug   quiet verbo debug      quiet verbo debug
 0  False False False   False False False  ->  False False False
 1   True False False   False False False  ->   True False False
 2  False  True False   False False False  ->  False  True False
 3   True  True False   False False False  ->  False False False
 4  False False  True   False False False  ->  False  True  True
 5   True False  True   False False False  ->  False  True  True
 6  False  True  True   False False False  ->  False  True  True
 7   True  True  True   False False False  ->  False  True  True
 8  False False False    True False False  ->   True False False
 9   True False False    True False False  ->   True False False
10  False  True False    True False False  ->   True False False
11   True  True False    True False False  ->   True False False
12  False False  True    True False False  ->   True False False
13   True False  True    True False False  ->   True False False
14  False  True  True    True False False  ->   True False False
15   True  True  True    True False False  ->   True False False
16  False False False   False  True False  ->  False  True False
17   True False False   False  True False  ->  False  True False
18  False  True False   False  True False  ->  False  True False
19   True  True False   False  True False  ->  False  True False
20  False False  True   False  True False  ->  False  True False
21   True False  True   False  True False  ->  False  True False
22  False  True  True   False  True False  ->  False  True False
23   True  True  True   False  True False  ->  False  True False
24  False False False    True  True False  ->  False False False
25   True False False    True  True False  ->  False False False
26  False  True False    True  True False  ->  False False False
27   True  True False    True  True False  ->  False False False
28  False False  True    True  True False  ->  False False False
29   True False  True    True  True False  ->  False False False
30  False  True  True    True  True False  ->  False False False
31   True  True  True    True  True False  ->  False False False
32  False False False   False False  True  ->  False  True  True
33   True False False   False False  True  ->  False  True  True
34  False  True False   False False  True  ->  False  True  True
35   True  True False   False False  True  ->  False  True  True
36  False False  True   False False  True  ->  False  True  True
37   True False  True   False False  True  ->  False  True  True
38  False  True  True   False False  True  ->  False  True  True
39   True  True  True   False False  True  ->  False  True  True
40  False False False    True False  True  ->  False  True  True
41   True False False    True False  True  ->  False  True  True
42  False  True False    True False  True  ->  False  True  True
43   True  True False    True False  True  ->  False  True  True
44  False False  True    True False  True  ->  False  True  True
45   True False  True    True False  True  ->  False  True  True
46  False  True  True    True False  True  ->  False  True  True
47   True  True  True    True False  True  ->  False  True  True
48  False False False   False  True  True  ->  False  True  True
49   True False False   False  True  True  ->  False  True  True
50  False  True False   False  True  True  ->  False  True  True
51   True  True False   False  True  True  ->  False  True  True
52  False False  True   False  True  True  ->  False  True  True
53   True False  True   False  True  True  ->  False  True  True
54  False  True  True   False  True  True  ->  False  True  True
55   True  True  True   False  True  True  ->  False  True  True
56  False False False    True  True  True  ->  False  True  True
57   True False False    True  True  True  ->  False  True  True
58  False  True False    True  True  True  ->  False  True  True
59   True  True False    True  True  True  ->  False  True  True
60  False False  True    True  True  True  ->  False  True  True
61   True False  True    True  True  True  ->  False  True  True
62  False  True  True    True  True  True  ->  False  True  True
63   True  True  True    True  True  True  ->  False  True  True