exchange: replace a "not x in ys" by more Pythonic "x not in ys" Found by one of our (Google-)internal tools. Differential Revision: https://phab.mercurial-scm.org/D7546

packaging: include defaultrc/*.rc instead of default.d/*.rc Differential Revision: https://phab.mercurial-scm.org/D7555

merge with stable

py3: make a pycompat.osdevnull, use it in extdiff Differential Revision: https://phab.mercurial-scm.org/D7545

subrepo: use pycompat.open directly instead of importing open from pycompat Differential Revision: https://phab.mercurial-scm.org/D7544

rust-dirs: address failing tests for `dirs` impl with a temporary fix https://phab.mercurial-scm.org/D7252 (5d40317d42b7083b49467502549e25f144888cb3) introduced a regression in Rust tests. This is a temporary fix that replicates the behavior of the C and Python impl, pending the resolution of the discussion (in the phabricator link) about how we actually want to solve this problem. Differential Revision: https://phab.mercurial-scm.org/D7503

cleanup: fix docstring formatting This is just removing the b'' prefix (except demandimportpy2), and making sure it is triple quoted. I skipped the mapping.py module in zope because that's 3rd party code. Differential Revision: https://phab.mercurial-scm.org/D7539

copies: split the combination of the copies mapping in its own function In some case, this part take up to 95% of the copy tracing that take about a hundred second. This poor performance comes from the fact we keep duplciating and merging dictionary that are mostly similar. I want to experiment with smarter native code to do this, so I need to isolate the function first.

copies: do not initialize the dictionary with root in changeset copies We don't strictly need the dict to be prefiled with `{}`. Not doing so will make the next changeset simpler. This part of a preparation to use native code for this part of copy tracing.

copies: expand `_chain` variable name to make the function easier to read This came up while explaining what the function is about. I find the function easier to follow that way.

copies: show we can read data from the "first" mapping In practice this make no difference since `t` is a copy of `a`. Having this change before hand will make the next cleanup changeset clearer. This is work on the road to using some native code for some performance critical part of copy tracing.

copies: return consistent type from revinfo The return is expected to be a list, so we should return a list. This make the use of this function from type-checked language (eg: rust) much simpler.

dateutil: correct default for Ymd in parsedate The code uses `0` for the default value of Ymd (year, month, and day), which seems suboptimal. For example, these will fail to parse: dateutil.parsedate('2000', formats=dateutil.extendeddateformats) dateutil.parsedate('Jan 2000', formats=dateutil.extendeddateformats) Fix it by providing sane defaults (1 instead of 0) for year, month, and day. The suboptimal behavior was introduced by 91bc001a592 (2010-12-29, "date: fix matching of underspecified date ranges"), which does not seem to justify the current behavior. Note end-users should not notice the subtle issue, because there are no formats in `defaultdateformats` that allow an explicit year with omitted month, or an explicit month with omitted day. Differential Revision: https://phab.mercurial-scm.org/D7520

test-doctest: include dateutil `mercurial.utils.dateutil` has dostrings that contain doctests. Include them. Differential Revision: https://phab.mercurial-scm.org/D7519

revlog: fix revset in reachableroots docstring `reachableroots` will only return a subset of `roots` when `includepath` is False. For example, given the following linear DAG: 2 | 1 | 0 Using roots=0+2, heads=1, the definition in the docstring does not match what `reachableroots` actually does: ipdb> repo.changelog.reachableroots(0, roots=[0,2],heads=[1]) [0] ipdb> repo.revs('heads(::(0+2) & (0+2)::1)') <baseset+ [1]> The fix is to do `heads & ::roots` (or `heads & heads::roots`) first, then select their ancestors: ipdb> repo.revs('heads(::((0+2) & (0+2)::1))') <baseset+ [0]> The docstring was introduced by fd92bfbbe02d9 (2015-06-19 "revset: rename revsbetween to reachableroots and add an argument"), which introduced the `includepath=False` behavior for graphlog grandparents use-case. I believe the docstring instead of the code should be changed because changing the code to match the docstring can result in suboptimal graphlog like: o :\ : o : : :/ o As opposite to the current "linearized" graphlog: o | o : o Differential Revision: https://phab.mercurial-scm.org/D7518

lock: pass "success" boolean to _afterlock callbacks This lets the callback decide if it should actually run or not. I suspect that most callbacks (and hooks) *should not* run in this scenario, but I'm trying to not break any existing behavior. `persistmanifestcache`, however, seems actively dangerous to run: we just encountered an exception and the repo is in an unknown state (hopefully a consistent one due to transactions, but this is not 100% guaranteed), and the data we cache may be based on this unknown state. This was observed by our users since we wrap some of the functions that persistmanifestcache calls and it expects that the repo object is in a certain state that we'd set up earlier. If the user hits ctrl-c before we establish that state, we end up crashing there. I'm going to make that extension resilient to this issue, but figured it might be a common issue and should be handled here as well instead of just working around the issue. Differential Revision: https://phab.mercurial-scm.org/D7459

relnotes: add note about changes to match.{explicit,reverse}dir Differential Revision: https://phab.mercurial-scm.org/D7508

graphlog: change state dict to attr struct This should help static analysis.

status: fix default value of status struct The default argument isn't overloaded. Before, the default constructor would create a struct having 7 list type objects.

typing: fix return type of logcmdutil.getrevs() Fixes the following errors: Invalid type annotation "'Tuple[smartset.BaseSet, changesetdiffer]'" [invalid-annotation] No attribute 'BaseSet' on module 'mercurial.smartset' getrevs: bad option in return type [bad-return-type] Expected: Tuple[mercurial.smartset.abstractsmartset, changesetdiffer] Actually returned: Tuple[mercurial.smartset.baseset, None]

typing: consolidate "if not globals():" trick Removes redundant inline comments. I think pycompat is good place to host this kind of constants.

rust-cpython: do not convert warning pattern to utf-8 bytes On Unix, both Rust Path and Mercurial expect a locale-dependent bytes, and we don't support Windows yet.

rust-cpython: import utils::files::* function at module level IIRC, it's common in Rust to call functions with the module prefix.

exchange: guard against method invocation on `b2caps=None` args I couldn't figure out how these are called, but the value is pretty obviously set at least for the cases that have tests. Differential Revision: https://phab.mercurial-scm.org/D7512

exchange: eliminate some bytes.format() calls Differential Revision: https://phab.mercurial-scm.org/D7511

windows: suppress pytype warnings for Windows imports and functions This should allow the modules to not be excluded, and not generate complaints on non Windows platforms. Differential Revision: https://phab.mercurial-scm.org/D7510

changectx: mark the parents of the working copy as non filtered If we successfully accessed the working copy, its parents are not filtered. Differential Revision: https://phab.mercurial-scm.org/D7491

repoview: add an explicit set of all filter that show the wc parents The `visible` set will always show the working copy parents. We record this in a specific set. This will allow to fast path some access. Differential Revision: https://phab.mercurial-scm.org/D7490

localrepo: introduce a `_quick_access_changeid` property Having faster access to `null` is cute… but limited. We want to speedup access to more useful revision, like `.`. We start with turning the fast path for `null` into something more generic. Differential Revision: https://phab.mercurial-scm.org/D7488

changectx: use unfiltered changelog to access parents of unfiltered revs If a revision is not filtered, we know that its parents are not either. So we can take a shortcut. This shortcut avoid the computation of all filtered revs in some cases. Differential Revision: https://phab.mercurial-scm.org/D7487

locarepo: also fastpath `nullid` lookup in __getitem__ We already use that fastpath for `"null"` and `nullrev`, using it for `nullid` is similar. Differential Revision: https://phab.mercurial-scm.org/D7486

repoview: add more tests to track operation not supposed to trigger filtering This test is useful to confirm we removed filtering trigger and to prevent it to come back without us noticing. The commands tested in the test were initially introduced one by one. However, on Martin von Zweigbergk request, we are adding them all at once. Differential Revision: https://phab.mercurial-scm.org/D7514

localrepo: mark nullrev has never filtered All repository have a null, and it cannot be filtered. Differential Revision: https://phab.mercurial-scm.org/D7484

changectx: add a "maybe filtered" filtered attribute There are changeset that we know not to be filtered (eg: `null`). In this case, we could access some information without triggering changelog filtering. We add an attribute to changectx that track this property. Differential Revision: https://phab.mercurial-scm.org/D7483

repoview: add a test that access actual changeset data of `null` Ideally, we would not trigger filtering here. However some work needs to happens first. Differential Revision: https://phab.mercurial-scm.org/D7482

localrepo: recognize trivial "null" queries in `anyrev` Bypassing the revset logic for trivial "null" queries means we can avoid to trigger the filtering logic in some cases. Differential Revision: https://phab.mercurial-scm.org/D7481

localrepo: also fastpath `nullrev` in __getitem__ As explained earlier, nullrev will exist in all repository, we do not need any special checking. Differential Revision: https://phab.mercurial-scm.org/D7480

repoview: add a test to track operation not supposed to trigger filtering This test will be useful to confirm we removed filtering trigger and to prevent it to come back without us noticing. Differential Revision: https://phab.mercurial-scm.org/D7479

repoview: display stack trace along side the debug message When a filter computation is triggered, If --traceback is provided, we will display a traceback in addition to the debug message. Differential Revision: https://phab.mercurial-scm.org/D7478

util: add an optional `prefix` argument to debugstacktrace This is useful when using it in a specific context. Differential Revision: https://phab.mercurial-scm.org/D7477

repoview: add a 'devel.debug.repo-filter' option If set, there will be debug message when a filter computation is triggered. This is going to be useful to remove various filtering trigger and to test they do not get reintroduced. Differential Revision: https://phab.mercurial-scm.org/D7476

localrepo: extract handling of some special value in __getitem__ The value "tip" should always be accessible, otherwise there is some problematic bug in the lower level. So we can access this outside of the general try/catch. If it fails some wider and actual big is in play. Differential Revision: https://phab.mercurial-scm.org/D7475

localrepo: extract handling of some special value in __getitem__ The value "null" will always be present in a repository. So this lookup should always succeed and do not need to be in the general try/catch. Differential Revision: https://phab.mercurial-scm.org/D7474

localrepo: add some basic comment for block in __getitem__ There are different early processing before getting to the core of the function. We highlight that fact. Differential Revision: https://phab.mercurial-scm.org/D7473

windows: further build fixes for the WiX installer With these fixes in place, the .msi actually builds for me again. Differential Revision: https://phab.mercurial-scm.org/D7509

cleanup: update references to /help/ that should now be /helptext/ I noticed this only because the WiX build was broken after 2e017696181f. I haven't tested this change with WiX, but it seems pretty obvious by inspection this should fix it. Differential Revision: https://phab.mercurial-scm.org/D7472

merge with stable

tests: byteify a few more things in simplestorerepo.py Differential Revision: https://phab.mercurial-scm.org/D7471

profiling: add a missing argument to the ProgrammingError constructor Differential Revision: https://phab.mercurial-scm.org/D7470

profiling: disable the import-error warning for the flamegraph module The module being missing is properly handled, but pytype was complaining. Differential Revision: https://phab.mercurial-scm.org/D7469

phases: make `allphases` a list on py3 also Pytype complained that `range` doesn't support slicing, so make this a list like it was on py2. Differential Revision: https://phab.mercurial-scm.org/D7468

wireprotov1server: capture Abort type before accessing the `hint` attribute Although the previous code worked, pytypes complained because `exc` is caught above as `BundleValueError, Abort, PushRaced`, and the other two don't have this attribute. Differential Revision: https://phab.mercurial-scm.org/D7463

tests: add more tests for "hg shelve --delete" It appears that the only tests for "hg shelve --delete" concern command errors (e.g. incompatible command options). Adding some more to check that non-existent names are handled and a success case. Differential Revision: https://phab.mercurial-scm.org/D7460

scmutil: add assertions to help pytype Differential Revision: https://phab.mercurial-scm.org/D7456

revset: add an assertion to help pytype Differential Revision: https://phab.mercurial-scm.org/D7455

import: add a --secret option Similarly to "hg commit", we add a --secret option to "hg import" for committing with the secret phase. The option has no short form since there already is a "-s" for "--similarity". .. feature:: ``hg import`` has a new --secret option for committing with the secret phase.

hgweb: add diffs to the json changeset template While it's possible to get the diff for each file individually via an obscure url that I'm not even sure is documented, (e.g. diff/{rev}/{filename}?style=json ), it is more convenient to provide the full diff from the changeset view. This is already normally computed for other non-JSON templates, so it seems like an oversight to omit it for the JSON one. The output format is a bit unwieldy but it's consistent with the existing format output by the aforementioned obscure url.

hgweb: add files to the json changeset template As far as I can tell, it's currently impossible via JSON to figure out what files are touched by a particular revision. Reusing the filenodelink and filenolink templates from the other hgweb templates should provide this information.

hgweb: add a status property to file list context The web templates merely indicate if files touched by this revision are in the commit or not, i.e. if they are removed. It would be helpful to have more context and also indicate whether the files are added, modified, or removed.

match: remove explicitdir attribute No one sets it or reads it anymore. Differential Revision: https://phab.mercurial-scm.org/D7441

dirstate: stop caring about match.explicitdir No one passes it in anymore. Differential Revision: https://phab.mercurial-scm.org/D7440

dirstate: include explicit matches in match.traversedir calls The caller should not have to register to get both explicit and non-explicit matches. Differential Revision: https://phab.mercurial-scm.org/D7439

commit: drop unused "vdirs" argument from repo.checkcommitpatterns() Differential Revision: https://phab.mercurial-scm.org/D7438

commit: rewrite check for `hg ci <path>` being a directory The matcher API is complicated by match.bad, match.explicitdir, and match.traversedir. We already have very few users of match.explicitdir. By rewriting this check we get close to being able to remove match.explicitdir. This may make the check slower, but I think that will be very marginal. Disclosure: I actually wrote this patch to solve a bug we've seen with an internal extension. The internal extension overrides the dirstate walk to only walk the files that our FUSE tells us are modified. That led to "vdirs" not getting populated as this code expected. I have wanted to get rid of match.explicitdir for a very long time, though. Differential Revision: https://phab.mercurial-scm.org/D7437

lock: fix race in lock-breaking code With low frequency, I see hg pulls fail with output like: abort: no such file or directory: .hg/store/lock I think what happens is, in lock.py, in: def _testlock(self, locker): if not self._lockshouldbebroken(locker): return locker # if locker dead, break lock. must do this with another lock # held, or can race and break valid lock. try: with lock(self.vfs, self.f + b'.break', timeout=0): self.vfs.unlink(self.f) except error.LockError: return locker if a lock is breakable on disk, and two hg processes concurrently get to the "if locker dead" comment, a possible interleaving is: process1 finishes executing the function and then process2 finishes executing the function. If that happens, process2 will either get ENOENT in self.vfs.unlink (resulting in the spurious failure above), or break a valid lock and potentially cause repository corruption. The fix is simple enough: make sure the lock is breakable _inside_ the critical section, because only then can we know that no other process can invalidate our knowledge on the lock on disk. I don't think there are tests for this. I've tested this manually with: diff --git a/mercurial/lock.py b/mercurial/lock.py --- a/mercurial/lock.py +++ b/mercurial/lock.py @@ -351,6 +351,8 @@ class lock(object): if not self._lockshouldbebroken(locker): return locker + import random + time.sleep(1. + random.random()) # if locker dead, break lock. must do this with another lock # held, or can race and break valid lock. try: @@ -358,6 +360,7 @@ class lock(object): self.vfs.unlink(self.f) except error.LockError: return locker + time.sleep(1) def testlock(self): """return id of locker if lock is valid, else None. and I see this change of behavior before/after this commit: $ $hg init repo $ cd repo $ ln -s $HOSTNAME/effffffc:987654321 .hg/wlock $ touch a $ $hg commit -Am_ & $hg commit -Am _; wait -abort: No such file or directory: '/tmp/repo/.hg/wlock' adding a +warning: ignoring unknown working parent 679a8959a8ca! +nothing changed Differential Revision: https://phab.mercurial-scm.org/D7199

lock: refactor in preparation for next commit Differential Revision: https://phab.mercurial-scm.org/D7198

extensions: suppress a pytype failure due to a typeshed bug Bug filed upstream, suppress the failure here so we can move on. Differential Revision: https://phab.mercurial-scm.org/D7410

dispatch: add some assertions to give pytype a helping hand Differential Revision: https://phab.mercurial-scm.org/D7409

extensions: hide two confusing import statements from pytype Differential Revision: https://phab.mercurial-scm.org/D7408

debugcommands: add assertions to convince pytype peer is not None This function is moderately annoyingly defined, and peer is set up iff we're not in raw-proto mode. That's fine, but it confuses pytype. Adding these assertions is a low-overhead way to convince pytype we're doing reasonable things. Differential Revision: https://phab.mercurial-scm.org/D7407

debugcommands: suppress import errors for pytype Differential Revision: https://phab.mercurial-scm.org/D7385

state: add a pytype annotation Differential Revision: https://phab.mercurial-scm.org/D7383

logcmdutil: add a type annotation This wasn't actually required in the end, as there was a real bug found by pytype, but the annotation helped me figure that out. We can drop this patch if that's the preference. Differential Revision: https://phab.mercurial-scm.org/D7382

cmdutil: add a pytype annotation to help out some callsites I think we could constrain the values here more than Any, but this lets us move forward with typechecking commands.py. Differential Revision: https://phab.mercurial-scm.org/D7381

hghave: add a check for the `xz` compression utility This isn't install by default on Mac, which causes a test failure. The logic for avoiding the command is a little goofy, but nested `#if` isn't supported, and it still seems worth running the hg command to see if anything explodes. With this, the py3 tests run (almost) cleanly on 10.14.6: # Ran 835 tests, 58 skipped, 1 failed. Alas, the mac-packaging test is skipped because it's slow. The failure here is in test-releasenotes-merging.t, complaining about not being able to import the `fuzzywuzzy` module. I have it installed on py3 (thus the test isn't skipped), but not on py2. So there must be some unintended cross pollination here when running `hg` commands. Differential Revision: https://phab.mercurial-scm.org/D7374

py3: use '%d' for formatting a revnum in `hg co --date` code Differential Revision: https://phab.mercurial-scm.org/D7429

py3: send bytes from Rust-created warning patterns Python code expects bytes in both Python 2 and Python 3, so we should send bytes. Differential Revision: https://phab.mercurial-scm.org/D7454

py3: pass bytes to `configint` and `configbool` Both functions require bytes, even in Python 3. Differential Revision: https://phab.mercurial-scm.org/D7453

templates: make {indent("", " ")} be empty indent() is documented to indent all non-empty lines, but it made an exception for the first line, which always got indented. I also made indent() not indent the first line even if an indent override was given for the first line. I think that is what one would usually want. Differential Revision: https://phab.mercurial-scm.org/D7432

tests: document behavior of indent() with empty first line indent() is documented to indent all non-empty lines, but it still indents the first line even if it's empty. Let's have tests for that. Differential Revision: https://phab.mercurial-scm.org/D7431

tests: check for Windows line endings We shouldn't generally be using Windows line endings in files under version control. I've accidentally committed a few files with Windows line endings recently. So let's add a test for this. Differential Revision: https://phab.mercurial-scm.org/D7448

tests: verify that Python scripts are blackened Scripts are Python too. They should be blackened. (As suggested by Pierre-Yves in D7420.) Differential Revision: https://phab.mercurial-scm.org/D7447

black: blacken scripts These scripts weren't blackened. I found these as part of adding script checking to test-check-format.t. # skip-blame black Differential Revision: https://phab.mercurial-scm.org/D7446

doc: don't use mutable default arguments It appears our mutable default arguments checker doesn't find arguments not on the first line of a function definition :( The arguments aren't used, so the default value is irrelevant. I found this when blackening this script in a future commit. Differential Revision: https://phab.mercurial-scm.org/D7445

tags: don't check for clean files when checking for dirty working copy We didn't request status for clean files so there should never be any entries in the list of clean files (at least not since dd773340a085), but it's misleading to check the list. Differential Revision: https://phab.mercurial-scm.org/D7428

branchmap: make "closed" a set from beginning instead of converting from list Differential Revision: https://phab.mercurial-scm.org/D7427

debugextensions: gracefully handle missing __file__ attributes This was crashing PyOxidizer. Differential Revision: https://phab.mercurial-scm.org/D7452

debuginstall: gracefully handle missing __file__ attributes This was crashing PyOxidizer. While here, point "Python lib" and "installed modules" to the oxidized binary when read from memory instead of pretending their location is unknown. Differential Revision: https://phab.mercurial-scm.org/D7451

typing: add pseudo localstr.__init__() to help pytype Apparently, pytype failed to parse localstr.__new__()? This fixes the following errors: line 126, in __hash__: No attribute '_utf8' on localstr [attribute-error] line 188, in tolocal: Function localstr.__init__ was called with the wrong arguments [wrong-arg-types] Expected: (self, string: str, ...) Actually passed: (self, string: bytes, ...)

typing: fix argument type of encoding.localstr() It takes both utf-8 and local strings in bytes type.

typing: suppress error of py2 encoding.strtolocal() and .strfromlocal() I don't know how to conditionally get rid of these py2/py3 overloads from .pyi file. Instead, this patch makes pytype ignore the false-positives: line 271, in strtolocal: bad option in return type [bad-return-type] Expected: bytes Actually returned: str

typing: constrain argument/return types of encoding.toutf8b()

typing: fix argument type of encoding.tolocal() and .fromutf8b() Fixes various pytype errors.

typing: fix forward reference in _Tlocalstr type bound AFAIK, a quoted string in type position is translated to its ForwardRef. https://github.com/python/typing/issues/34 This fixes the following error: File "mercurial/encoding.py", line 38, in <module>: Name 'localstr' is not defined [name-error]

encoding: make getcols() raise exception explicitly This helps pytype. File "mercurial/encoding.py", line 329, in getcols: bad option in return type [bad-return-type] Expected: bytes Actually returned: None

setup: conditionalize access to `sys.dllhandle` when building extensions This code is only run on Windows, and was crashing PyOxidizer when running in `setup-py-install` mode. Now an oxidized binary can be built by simply pointing to setup.py. Something is slightly different now that it's not being built from a virtualenv. Previously, `hg version` could print to the screen, but now it aborts saying "Incorrect function". But I can see the output if redirected to a file, and it's not complaining about missing C extensions, so I think those are loading now (unlike from the virtualenv). The interesting this about this incorrect function output is that it failed when initially built. I then went back and did a `make clean` and `make local` with py3 and then py2 to ensure I didn't break the existing code. At that point I ran the oxidized executable again and it was able to print to the screen normally! So I ran `pyoxidizer build` again, it only output the following, and then running the executable failed to output again: (pyO2_venv) C:\Users\Matt\hg3\hg_pyO2>pyoxidizer build Finished dev [unoptimized + debuginfo] target(s) in 0.12s packaging application into C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug purging C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug copying C:/Users/Matt/hg3/hg_pyO2\build\target\x86_64-pc-windows-msvc\debug\hg_pyO2.exe to C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug\hg_pyO2.exe resolving packaging state... writing license for [...] hg_pyO2 packaged into C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug executable path: C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug\hg_pyO2.exe Differential Revision: https://phab.mercurial-scm.org/D7444