view tests/test-narrow-trackedcmd.t @ 43709:039fbd14d4e2

lock: fix race in lock-breaking code

With low frequency, I see hg pulls fail with output like:
abort: no such file or directory: .hg/store/lock

I think what happens is, in lock.py, in:

    def _testlock(self, locker):
        if not self._lockshouldbebroken(locker):
            return locker

        # if locker dead, break lock.  must do this with another lock
        # held, or can race and break valid lock.
        try:
            with lock(self.vfs, self.f + b'.break', timeout=0):
                self.vfs.unlink(self.f)
        except error.LockError:
            return locker

if a lock is breakable on disk, and two hg processes concurrently get
to the "if locker dead" comment, a possible interleaving is: process1
finishes executing the function and then process2 finishes executing
the function. If that happens, process2 will either get ENOENT in
self.vfs.unlink (resulting in the spurious failure above), or break a
valid lock and potentially cause repository corruption.

The fix is simple enough: make sure the lock is breakable _inside_ the
critical section, because only then can we know that no other process
can invalidate our knowledge on the lock on disk.

I don't think there are tests for this. I've tested this manually
with:

diff --git a/mercurial/lock.py b/mercurial/lock.py
--- a/mercurial/lock.py
+++ b/mercurial/lock.py
@@ -351,6 +351,8 @@ class lock(object):
         if not self._lockshouldbebroken(locker):
             return locker

+        import random
+        time.sleep(1. + random.random())
         # if locker dead, break lock.  must do this with another lock
         # held, or can race and break valid lock.
         try:
@@ -358,6 +360,7 @@ class lock(object):
                 self.vfs.unlink(self.f)
         except error.LockError:
             return locker
+        time.sleep(1)

     def testlock(self):
         """return id of locker if lock is valid, else None.

and I see this change of behavior before/after this commit:

 $ $hg init repo
 $ cd repo
 $ ln -s $HOSTNAME/effffffc:987654321 .hg/wlock
 $ touch a
 $ $hg commit -Am_ & $hg commit -Am _; wait
-abort: No such file or directory: '/tmp/repo/.hg/wlock'
 adding a
+warning: ignoring unknown working parent 679a8959a8ca!
+nothing changed

Differential Revision: https://phab.mercurial-scm.org/D7199

#testcases flat tree
  $ . "$TESTDIR/narrow-library.sh"

#if tree
  $ cat << EOF >> $HGRCPATH
  > [experimental]
  > treemanifest = 1
  > EOF
#endif

  $ hg init master
  $ cd master
  $ cat >> .hg/hgrc <<EOF
  > [narrow]
  > serveellipses=True
  > EOF

  $ mkdir inside
  $ echo 'inside' > inside/f
  $ hg add inside/f
  $ hg commit -m 'add inside'

  $ mkdir widest
  $ echo 'widest' > widest/f
  $ hg add widest/f
  $ hg commit -m 'add widest'

  $ mkdir outside
  $ echo 'outside' > outside/f
  $ hg add outside/f
  $ hg commit -m 'add outside'

  $ cd ..

narrow clone the inside file

  $ hg clone --narrow ssh://user@dummy/master narrow --include inside
  requesting all changes
  adding changesets
  adding manifests
  adding file changes
  added 2 changesets with 1 changes to 1 files
  new changesets *:* (glob)
  updating to branch default
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cd narrow
  $ hg tracked
  I path:inside
  $ ls
  inside
  $ cat inside/f
  inside
  $ cd ..

add more upstream files which we will include in a wider narrow spec

  $ cd master

  $ mkdir wider
  $ echo 'wider' > wider/f
  $ hg add wider/f
  $ echo 'widest v2' > widest/f
  $ hg commit -m 'add wider, update widest'

  $ echo 'widest v3' > widest/f
  $ hg commit -m 'update widest v3'

  $ echo 'inside v2' > inside/f
  $ hg commit -m 'update inside'

  $ mkdir outside2
  $ echo 'outside2' > outside2/f
  $ hg add outside2/f
  $ hg commit -m 'add outside2'

  $ echo 'widest v4' > widest/f
  $ hg commit -m 'update widest v4'

  $ hg log -T "{if(ellipsis, '...')}{rev}: {desc}\n"
  7: update widest v4
  6: add outside2
  5: update inside
  4: update widest v3
  3: add wider, update widest
  2: add outside
  1: add widest
  0: add inside

  $ cd ..

Testing the --import-rules flag of `hg tracked` command

  $ cd narrow
  $ hg tracked --import-rules
  hg tracked: option --import-rules requires argument
  hg tracked [OPTIONS]... [REMOTE]
  
  show or change the current narrowspec
  
  options ([+] can be repeated):
  
      --addinclude VALUE [+]       new paths to include
      --removeinclude VALUE [+]    old paths to no longer include
      --auto-remove-includes       automatically choose unused includes to
                                   remove
      --addexclude VALUE [+]       new paths to exclude
      --import-rules VALUE         import narrowspecs from a file
      --removeexclude VALUE [+]    old paths to no longer exclude
      --clear                      whether to replace the existing narrowspec
      --force-delete-local-changes forces deletion of local changes when
                                   narrowing
      --update-working-copy        update working copy when the store has
                                   changed
   -e --ssh CMD                    specify ssh command to use
      --remotecmd CMD              specify hg command to run on the remote side
      --insecure                   do not verify server certificate (ignoring
                                   web.cacerts config)
  
  (use 'hg tracked -h' to show more help)
  [255]
  $ hg tracked --import-rules doesnotexist
  abort: cannot read narrowspecs from '$TESTTMP/narrow/doesnotexist': $ENOENT$
  [255]

  $ cat > specs <<EOF
  > %include foo
  > [include]
  > path:widest/
  > [exclude]
  > path:inside/
  > EOF

  $ hg tracked --import-rules specs
  abort: including other spec files using '%include' is not supported in narrowspec
  [255]

  $ cat > specs <<EOF
  > [include]
  > outisde
  > [exclude]
  > inside
  > EOF

  $ hg tracked --import-rules specs
  comparing with ssh://user@dummy/master
  searching for changes
  looking for local changes to affected paths
  deleting data/inside/f.i
  deleting meta/inside/00manifest.i (tree !)
  saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-widen.hg (glob)
  adding changesets
  adding manifests
  adding file changes
  added 2 changesets with 0 changes to 0 files
  $ hg tracked
  I path:outisde
  X path:inside

Testing the --import-rules flag with --addinclude and --addexclude

  $ cat > specs <<EOF
  > [include]
  > widest
  > EOF

  $ hg tracked --import-rules specs --addinclude 'wider/'
  comparing with ssh://user@dummy/master
  searching for changes
  saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-widen.hg (glob)
  adding changesets
  adding manifests
  adding file changes
  added 3 changesets with 1 changes to 1 files
  $ hg tracked
  I path:outisde
  I path:wider
  I path:widest
  X path:inside

  $ cat > specs <<EOF
  > [exclude]
  > outside2
  > EOF

  $ hg tracked --import-rules specs --addexclude 'widest'
  comparing with ssh://user@dummy/master
  searching for changes
  looking for local changes to affected paths
  deleting data/widest/f.i
  deleting meta/widest/00manifest.i (tree !)
  $ hg tracked
  I path:outisde
  I path:wider
  X path:inside
  X path:outside2
  X path:widest

  $ hg tracked --import-rules specs --clear
  abort: the --clear option is not yet supported
  [255]

Testing with passing a out of wdir file

  $ cat > ../nspecs <<EOF
  > [include]
  > widest
  > EOF

  $ hg tracked --import-rules ../nspecs
  comparing with ssh://user@dummy/master
  searching for changes
  saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-widen.hg (glob)
  adding changesets
  adding manifests
  adding file changes
  added 3 changesets with 0 changes to 0 files

  $ cd ..

Testing tracked command on a non-narrow repo

  $ hg init non-narrow
  $ cd non-narrow
  $ hg tracked --addinclude foobar
  abort: the tracked command is only supported on repositories cloned with --narrow
  [255]