lock: fix race in lock-breaking code With low frequency, I see hg pulls fail with output like: abort: no such file or directory: .hg/store/lock I think what happens is, in lock.py, in: def _testlock(self, locker): if not self._lockshouldbebroken(locker): return locker # if locker dead, break lock. must do this with another lock # held, or can race and break valid lock. try: with lock(self.vfs, self.f + b'.break', timeout=0): self.vfs.unlink(self.f) except error.LockError: return locker if a lock is breakable on disk, and two hg processes concurrently get to the "if locker dead" comment, a possible interleaving is: process1 finishes executing the function and then process2 finishes executing the function. If that happens, process2 will either get ENOENT in self.vfs.unlink (resulting in the spurious failure above), or break a valid lock and potentially cause repository corruption. The fix is simple enough: make sure the lock is breakable _inside_ the critical section, because only then can we know that no other process can invalidate our knowledge on the lock on disk. I don't think there are tests for this. I've tested this manually with: diff --git a/mercurial/lock.py b/mercurial/lock.py --- a/mercurial/lock.py +++ b/mercurial/lock.py @@ -351,6 +351,8 @@ class lock(object): if not self._lockshouldbebroken(locker): return locker + import random + time.sleep(1. + random.random()) # if locker dead, break lock. must do this with another lock # held, or can race and break valid lock. try: @@ -358,6 +360,7 @@ class lock(object): self.vfs.unlink(self.f) except error.LockError: return locker + time.sleep(1) def testlock(self): """return id of locker if lock is valid, else None. and I see this change of behavior before/after this commit: $ $hg init repo $ cd repo $ ln -s $HOSTNAME/effffffc:987654321 .hg/wlock $ touch a $ $hg commit -Am_ & $hg commit -Am _; wait -abort: No such file or directory: '/tmp/repo/.hg/wlock' adding a +warning: ignoring unknown working parent 679a8959a8ca! +nothing changed Differential Revision: https://phab.mercurial-scm.org/D7199

lock: refactor in preparation for next commit Differential Revision: https://phab.mercurial-scm.org/D7198

extensions: suppress a pytype failure due to a typeshed bug Bug filed upstream, suppress the failure here so we can move on. Differential Revision: https://phab.mercurial-scm.org/D7410

dispatch: add some assertions to give pytype a helping hand Differential Revision: https://phab.mercurial-scm.org/D7409

extensions: hide two confusing import statements from pytype Differential Revision: https://phab.mercurial-scm.org/D7408

debugcommands: add assertions to convince pytype peer is not None This function is moderately annoyingly defined, and peer is set up iff we're not in raw-proto mode. That's fine, but it confuses pytype. Adding these assertions is a low-overhead way to convince pytype we're doing reasonable things. Differential Revision: https://phab.mercurial-scm.org/D7407

debugcommands: suppress import errors for pytype Differential Revision: https://phab.mercurial-scm.org/D7385

state: add a pytype annotation Differential Revision: https://phab.mercurial-scm.org/D7383

logcmdutil: add a type annotation This wasn't actually required in the end, as there was a real bug found by pytype, but the annotation helped me figure that out. We can drop this patch if that's the preference. Differential Revision: https://phab.mercurial-scm.org/D7382

cmdutil: add a pytype annotation to help out some callsites I think we could constrain the values here more than Any, but this lets us move forward with typechecking commands.py. Differential Revision: https://phab.mercurial-scm.org/D7381

hghave: add a check for the `xz` compression utility This isn't install by default on Mac, which causes a test failure. The logic for avoiding the command is a little goofy, but nested `#if` isn't supported, and it still seems worth running the hg command to see if anything explodes. With this, the py3 tests run (almost) cleanly on 10.14.6: # Ran 835 tests, 58 skipped, 1 failed. Alas, the mac-packaging test is skipped because it's slow. The failure here is in test-releasenotes-merging.t, complaining about not being able to import the `fuzzywuzzy` module. I have it installed on py3 (thus the test isn't skipped), but not on py2. So there must be some unintended cross pollination here when running `hg` commands. Differential Revision: https://phab.mercurial-scm.org/D7374

py3: use '%d' for formatting a revnum in `hg co --date` code Differential Revision: https://phab.mercurial-scm.org/D7429

py3: send bytes from Rust-created warning patterns Python code expects bytes in both Python 2 and Python 3, so we should send bytes. Differential Revision: https://phab.mercurial-scm.org/D7454

py3: pass bytes to `configint` and `configbool` Both functions require bytes, even in Python 3. Differential Revision: https://phab.mercurial-scm.org/D7453

templates: make {indent("", " ")} be empty indent() is documented to indent all non-empty lines, but it made an exception for the first line, which always got indented. I also made indent() not indent the first line even if an indent override was given for the first line. I think that is what one would usually want. Differential Revision: https://phab.mercurial-scm.org/D7432

tests: document behavior of indent() with empty first line indent() is documented to indent all non-empty lines, but it still indents the first line even if it's empty. Let's have tests for that. Differential Revision: https://phab.mercurial-scm.org/D7431

tests: check for Windows line endings We shouldn't generally be using Windows line endings in files under version control. I've accidentally committed a few files with Windows line endings recently. So let's add a test for this. Differential Revision: https://phab.mercurial-scm.org/D7448

tests: verify that Python scripts are blackened Scripts are Python too. They should be blackened. (As suggested by Pierre-Yves in D7420.) Differential Revision: https://phab.mercurial-scm.org/D7447

black: blacken scripts These scripts weren't blackened. I found these as part of adding script checking to test-check-format.t. # skip-blame black Differential Revision: https://phab.mercurial-scm.org/D7446

doc: don't use mutable default arguments It appears our mutable default arguments checker doesn't find arguments not on the first line of a function definition :( The arguments aren't used, so the default value is irrelevant. I found this when blackening this script in a future commit. Differential Revision: https://phab.mercurial-scm.org/D7445

tags: don't check for clean files when checking for dirty working copy We didn't request status for clean files so there should never be any entries in the list of clean files (at least not since dd773340a085), but it's misleading to check the list. Differential Revision: https://phab.mercurial-scm.org/D7428

branchmap: make "closed" a set from beginning instead of converting from list Differential Revision: https://phab.mercurial-scm.org/D7427

debugextensions: gracefully handle missing __file__ attributes This was crashing PyOxidizer. Differential Revision: https://phab.mercurial-scm.org/D7452

debuginstall: gracefully handle missing __file__ attributes This was crashing PyOxidizer. While here, point "Python lib" and "installed modules" to the oxidized binary when read from memory instead of pretending their location is unknown. Differential Revision: https://phab.mercurial-scm.org/D7451

typing: add pseudo localstr.__init__() to help pytype Apparently, pytype failed to parse localstr.__new__()? This fixes the following errors: line 126, in __hash__: No attribute '_utf8' on localstr [attribute-error] line 188, in tolocal: Function localstr.__init__ was called with the wrong arguments [wrong-arg-types] Expected: (self, string: str, ...) Actually passed: (self, string: bytes, ...)

typing: fix argument type of encoding.localstr() It takes both utf-8 and local strings in bytes type.

typing: suppress error of py2 encoding.strtolocal() and .strfromlocal() I don't know how to conditionally get rid of these py2/py3 overloads from .pyi file. Instead, this patch makes pytype ignore the false-positives: line 271, in strtolocal: bad option in return type [bad-return-type] Expected: bytes Actually returned: str

typing: constrain argument/return types of encoding.toutf8b()

typing: fix argument type of encoding.tolocal() and .fromutf8b() Fixes various pytype errors.

typing: fix forward reference in _Tlocalstr type bound AFAIK, a quoted string in type position is translated to its ForwardRef. https://github.com/python/typing/issues/34 This fixes the following error: File "mercurial/encoding.py", line 38, in <module>: Name 'localstr' is not defined [name-error]