test-https: properly conditionalize Windows vs non-Windows output There's a bug in the (feature !) annotation, and this needs to be resolved before fixing it. Once fixed, the glob '*/missing' will match the output '/missing' on non-Windows platforms, consuming the output line and leaving nothing for the no-windows required line. I probably should have made (foo !) match only the 'foo' feature, and be ignored otherwise instead of optional. (re) didn't work here, because the parenthesis need to be escaped too.

Added signature for changeset 857876ebaed4

Added tag 4.3-rc for changeset 857876ebaed4

merge default into stable for code freeze

releasenotes: add custom admonitions support for release notes By default, the extension has default sections like fix, feature, perf etc.. This patch allow user to add support for custom admonition. In order to add a custom admonition, one needs to have a .hgreleasenotes file inside the repository. All the custom directive with name specified under the tag [sections] will be usable by the extension. One important thing to keep in mind is if there exists any custom admonitions with same key as default then they will override the default ones.

check-config: mention the file and line of the error I used this to more quickly track down a failing test-check-config.t issue in another repo. I thought it might be useful more generally, so I'm sending it out in case others think it's a worthwhile change.

rebase: add config to move rebase into a single transaction This was previously landed as cf8ad0e6c0e4 but backed out in a5abaa81fa because it broke hook mid rebase and caused conflict resolution data loss in the event of unexpected exceptions. This new version adds the behavior back but behind a config flag, since the performance improvement is notable in large repositories. The next patch adds a test covering this config. The old commit message was: Previously, rebasing would open several transaction over the course of rebasing several commits. Opening a transaction can have notable overhead (like copying the dirstate) which can add up when rebasing many commits. This patch adds a single large transaction around the actual commit rebase operation, with a catch for intervention which serializes the current state if we need to drop back to the terminal for user intervention. Amazingly, almost all the tests seem to pass. On large repos with large working copies, this can speed up rebasing 7 commits by 25%. I'd expect the percentage to be a bit larger for rebasing even more commits. There are minor test changes because we're rolling back the entire transaction during unexpected exceptions instead of just stopping mid-rebase, so there's no more backup bundle. It also leave an unknown file in the working copy, since our clean up 'hg update' doesn't delete unknown files. (grafted from cca36c7f35261b0e31beb226bf361067ef0e06ab) (grafted from dc497d8705b71503e32e07bd33925c1e42cf9c9a) Differential Revision: https://phab.mercurial-scm.org/D134

run-tests: remove unnecessary 'with_color' variable Its value is always the same as that of 'pygmentspresent'. Differential Revision: https://phab.mercurial-scm.org/D118

run-tests: warn if --color=always and no pygments installed Differential Revision: https://phab.mercurial-scm.org/D117

run-tests: drop unnecessary check for pygmentspresent 2893face0af5 (run-tests: check if stream is a tty before using color, 2017-07-18) made the check redundant but forgot to remove it. Differential Revision: https://phab.mercurial-scm.org/D116

run-tests: pass color option via test case object , not global var At first I updated the color field of the 'options' object (from the CLI parser), but then I decided to put it directly on the test case object itself to avoid mutating the shared object (even though all tests would have the same value). Differential Revision: https://phab.mercurial-scm.org/D114

phabricator: sanity check Differential Revision from commit message Previously, we trust Differential Revision in commit message blindly. This patch adds sanity check so a host name change will be detected and the commit message will be ignored. Differential Revision: https://phab.mercurial-scm.org/D35

test-phases-exchange: properly conditionalize output I forgot to invert the conditions, and the test runner didn't catch it[1]. But since only '&&' is supported in the per-line feature tests, this won't work anyway. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-July/101941.html

commitextras: mark the extension as ADVANCED

run-tests: check if stream is a tty before using color Previous implementation (e80041832eec) checked only if sys.stderr was a tty which was less general. Also makes sure that colors is never used if pygments is not available, irrespective of --color flag value.

tests: rewrite test-rebase-detach.t using drawdag This makes it much clearer because each test case can have exactly the graph it wants (no extra cruft just because other tests want it). Differential Revision: https://phab.mercurial-scm.org/D94

tests: simplify config in test-rebase-detach.t Differential Revision: https://phab.mercurial-scm.org/D93

drawdag: include files from both parents in merge commits Consider a graph like this: D |\ B C |/ A drawdag will add a file called A in commit A, file B in B, file C in C. That's fine and expected. In merge commits like D, I would expect the files and their contents to be taken from the parent commits, so commit D in this example would have files A, B, and C. However, drawdag will instead add the file D compared to the first parent. Depending on whether B or C got a smaller nodeid, the contents of D would be {A, B, D} or {A, C, D}. This patch changes it to to be {A, B, C}. Differential Revision: https://phab.mercurial-scm.org/D92

gitweb: preserve whitespace in description Without this, multiple spaces or tabs in the commit message aren't preserved and things like tables don't align properly. As part of adding the CSS rule, we had to cuddle the content with the <div> to not introduce leading and trailing whitespace. The "addbreaks" filter was also removed because it would insert an additional newline, effectively double spacing content. Differential Revision: https://phab.mercurial-scm.org/D113

sparse: add a requirement when a repository uses sparse (BC) The presence of a sparse checkout can confuse legacy clients or clients without sparse enabled for reasons that should be obvious. This commit introduces a new repository requirement that tracks whether sparse is enabled. The requirement is added when a sparse config is activated and removed when the sparse config is reset. The localrepository constructor has been taught to not open repos with this requirement unless the sparse feature is enabled. It yields a more actionable error message than what you would get if the lockout were handled strictly at the requirements verification phase. Old clients that aren't sparse aware will see the generic "repository requires features unknown to this Mercurial" error, however. The new requirement has "exp" in its name to reflect the experimental nature of sparse. There's a chance that the eventual non-experimental feature won't change significantly and we could have squatted on the "sparse" requirement without ill effect. If that happens, we can teach new clients to still recognize the old name. But I suspect we'll sneak in some BC and we'll want a new requirement to convey new meaning. Differential Revision: https://phab.mercurial-scm.org/D110

sparse: consolidate common code for writing sparse config In 3 functions we were writing the sparse config and updating the working directory. In two of them we had a transaction-like process for restoring the sparse config in case of wdir update fail. Because the pattern is common, we've already made mistakes, and the complexity will increase in the near future, let's consolidate the code into a reusable function. As part of this refactor, we end up reading the "sparse" file twice when updating it. This is a bit sub-optimal. But I don't think it is worth the code complexity to pass around the variables to avoid the redundancy. Differential Revision: https://phab.mercurial-scm.org/D109

revset: pass repo when passing ui The repo instance is currently only used to provide a changeset lookup function as part of parsing revsets. I /think/ this allows node fragments to resolve. I'm not sure why we wouldn't want this to always "just work" if parsing a revset string. Plus, an upcoming commit will introduce a new consumer that needs a handle on the repo. So passing it more often will make that code work more. Passing a repo instance in all callers of revset.match* results in a bunch of test changes. Notably, branch and tags caches get populated as part of evaluating revsets. I'm not sure if this is desirable. So this patch takes the conservative approach and only passes the repo if we're passing a ui instance. Differential Revision: https://phab.mercurial-scm.org/D97

win32: copy-edit debugssl messages to match prevailing style

run-tests: make sure to check if pygments is installed before using it e80041832e introduced support to color the output of tests but used pygments without checking whether it's installed or not. That breaks test-run-tests.t for machines which don't have pygments installed. This patch conditionalize the color test in test-run-tests.t and also add a check to make sure pygments is installed before using that.

sparse: require [section] in sparse config files (BC) Previously, [include] was implicit and pattern lines before a [section] were added to includes. Because the format may change in the future and explicit behavior, well, more explicit, this commit changes the config parser to reject pattern lines that don't occur in a [section]. Differential Revision: https://phab.mercurial-scm.org/D96

sparse: use set for capturing profiles Order doesn't need to be preserved. A set is acceptable. Differential Revision: https://phab.mercurial-scm.org/D95

util: remove dead code which used to be for old python2 versions Differential Revision: https://phab.mercurial-scm.org/D107

status: add a flag to terse the output (issue4119) This adds an experimental flag -t/--terse which will terse the output. The terse flag will respect other flags which filters the output. The flag takes a string whose value can be a subsequence of "marduic" (the order does not matter here.) Ignored files are not considered while tersing unless -i flag is passed or 'i' is there is the terse flag value. The flag is experimental for testing as there may be cases which will produce strange results with the flag. We can set the terse on by default by simply passing 'u' to the cmdutil.tersestatus(). This patch also adds a test file with tests covering the new feature.

commitextras: check the format of the arguments and no internal key is used This patch adds check to make the arguments are passed as KEY=VALUE and no key which is used internally is passed. This patch also adds test for the extension.

commitextras: move fb extension to core which add extras to a commit This patch moves the Facebook extension to add extra fields to a commit to a in-core extension.

archive: add an experimental config to control the metadata file template Experimental because given the possible complexity, it may be worth figuring out how to load this from a file, similar to the style files for the log command, instead of trying to stuff it on the command line.

archive: use a templater to build the metadata file There are no visible changes here. I'm starting to wonder if adding the '+' to the 'node' line instead of a separate key line in 3047167733dc was the right thing to do. The '{node}' keyword never includes '+' elsewhere, and the way setup.py works, it would truncate it anyway. Additionally, the file is missing '{p2node}' when 'wdir()' merges are archived. I thought about adding an 'identify' line that would correspond to `hg id -n`. But the other nodes are the full 40 characters, and the output most useful for versioning is the short form. All of this cries out for customization via templating. (Although maybe having the short identify line by default is still a good idea.)

commandserver: do not handle EINTR for selector.select selectors2 library handles EINTR transparently so we don't need to handle select.error ourselves.

debugobsolete: also report the number of obsoleted changesets This seems useful to have the number of obsoleted changesets when calling debugobsolete.

transaction-summary: display the summary for all transactions Now that we records "all" changes happening in a transaction (in tr.changes) we will be able to provide better report on various changes (phases turned public, changeset obsoleted, branch merged or created, etc..) This is far too late in the cycle to play with this, but having this existing method called more widely will help extensions to play around with various options during the 4.4 cycle. Instead of calling registersummarycallback only for transactions we want, we always call it and use the transaction name to decide when to report (eg: we do not want `hg amend` to report new obsoleted changesets). Filtering on transaction name does not seems great, but seems good enough for the moment. We can change the API during the next cycle. The previous manual call during unbundling of the bundle2 "obsmarkers" part is no longer necessary and has been dropped.

test-strip: fix call to super class in crashstriprepo The 'self' argument is already taken care of by the 'super(...)' call. Instead we were passing the repository instance as the 'desc' value. This got discovered by breaking the next changeset.

share: share 'cachevfs' with the source clone (issue5108) Share extension now also share caches reads and writes. Not sharing caches results in costly caches recomputations which can takes up to minutes when using shares on large repositories. There are a couple of file in the '.hg/cache/' that depends of the current visibility. Visibility can be affected by the working copy location, something which is specific to each share. We ignores them for this series because they: * are the minority, * already have a good fallback to other precomputed caches, * are only affected when people use the experimental evolution feature.

cachevfs: add a devel warning for cache access though 'vfs' This will help third party extensions to migrate to the new 'cachevfs'.

cachevfs: migration the tags fnode cache to 'cachevfs' This will help sharing the cache between shares.

cachevfs: migrate tagscache to 'cachevfs' This will help sharing the cache between shares.

cachevfs: migration the revbranchcache to 'cachevfs' This will help sharing the cache between shares.

cachevfs: use the new vfs in when computing branchmap cache This will help sharing the cache between shares.

cachevfs: add a vfs dedicated to cache Most of the cache content lives in '.hg/cache/'. Moreover they are computed exclusively from data in the '.hg/store' directory. This creates issues with the share extension as the '.hg/store' directory is shared but the '.hg/cache' is not. On large repositories, this makes this prevent some usage of the share extension inefficient as some caches can take minutes to be recomputed. To improve the situation, we introduce a new 'cachevfs' that will be dedicated to cache reading and writing. In the next patches of this series, we'll migrate the 4 existing caches to it and update the share extension.

vfsward: register 'write with no lock' warnings as 'check-locks' config Update 'write with no lock' warnings in order to be better controlled by the config. We reuse the option used for lock order for these other lock related message. The message can now be disabled using 'devel.check-locks = no' (in addition to the usual 'devel.all-warnings = no').

demandimport: prefer loaded module over package attribute (issue5617) In general, the attribute of the same name is overwritten by executing an import statement. import a.b print(a.b.c) # 'c' of a/b/__init__.py from a.b.c import d print(a.b.c) # a/b/c.py However, this appears not true for the scenario described in the test case, and surprisingly, "from a.b.c import d" works even if "a.b.c" is not a module. This patch works around the problem by taking the right module from sys.modules if available.

demandimport: drop Py3 workarounds from Py2 implementation

demandimport: drop hack for old Pythons which had no level argument _import() was added by 868282fa29d8, but it is identical to _origimport() now.

tests: drop "absimport" conditionals which should be always true

color: drop the now useless color extension all the extension features are provided by core since 4.2.

extensions: expand the builtins extensions declaration This will future updates of the set cleaner and more readable.

debugextensions: drop the color from the test We are about to remove the extension so we remove trace of it in this specific test to prevent it to be confused.

configitems: register the 'bugzilla.notify' config

configitems: register the 'ui.mergemarkertemplate' config

configitems: register the 'ui.color' config

color: drop the _enabledbydefault module variable Since color is on by default, cleanup the now useless variable in both core and color extension.

configitems: register the 'ui.forcecwd' config

configitems: register the 'ui.fallbackencoding' config

bookmark: deprecate direct del of a bookmark value We want all bookmark deletion to go through 'applychanges', so lets deprecate legacy ways of doing bookmark deletion.

bookmark: deprecate direct set of a bookmark value We want all bookmark update to go through 'applychanges', so lets deprecate legacy ways of doing bookmark update.

bookmark: track bookmark changes at the transaction level The transaction has now a 'bookmarks' dictionary in tr.changes. The structure of the dictionary is {BOOKMARK_NAME: (OLD_NODE, NEW_NODE)}. If a bookmark is deleted NEW_NODE will be None. If a bookmark is created OLD_NODE will be None. If the bookmark is updated multiple time, the initial value is preserved.

bookmark: deprecate 'recordchange' in favor of 'applychanges' Now that we have migrated all in-core caller of 'recordchange' to 'applychanges', deprecate 'recordchange' so external callers will move to the new unified method.

bookmark: drop deletedivergent It has no caller anymore.

bookmark: use 'divergent2delete' in checkconflict checkconflict used to also do some bookmark deletion in case of divergence. It is a bit suspicious given the function name, but it's not the goal of this series. In order to unify bookmarks changing, checkconflict now return the list of divergent bookmarks to clean up and the callers must clean them by calling applyphases.

bookmark: use 'divergent2delete' when updating a bookmark

bookmark: use 'divergent2delete' in 'scmutil.cleanupnode'

bookmark: split out target computation from 'deletedivergent' We want to use applychanges in order to unify bookmark movement. We need a way to compute divergence deletion without actually removing them. We split the function in two in this patch while we migrate the existing users of this code on next patches.

bookmark: remove a useless 'recordchange' in the amend code We do not touch the bookmarks store in this code, just the active bookmark, not covered by the transaction. So it seems we can safely drop this call and the tests agree with us.

test-phase-exchange: stabilize for Windows

debugignore: eliminate inconsistencies with `hg status` (issue5222) Using a matcher for this command allows processing the named file(s) as relative to cwd. It also leverages the icasefs normalization logic the same way the status command does. (However, a false indicator is given for a nonexistent file in some cases, e.g. passing 'foo.REJ' when that file doesn't exist, and the rule is '*.rej'. Maybe the regex itself needs to be case insensitive on these platforms, at least for the debug command.) Finally, the file printed is relative to cwd and uses platform specific slashes, so a few (glob)s were needed in seemingly unrelated tests.

commandserver: close selector explicitly The selector does not have a __del__ method and needs a manual close. We can also use "with selector" but that makes the code too indented. Therefore append a "selector.close()" after the end of the main loop for now.

scmutil: remove duplicated import of i18n._()

obsstore: let read marker API take a range of offsets This allows us to read a customized range of markers, instead of loading all of them. The condition of stop is made consistent across C and Python implementation so we will still read marker when offset=a, stop=a+1.

commandserver: use selectors2 Previously, commandserver was using select.select. That could have issue if _sock.fileno() >= FD_SETSIZE (usually 1024), which raises: ValueError: filedescriptor out of range in select() We got that in production today, although it's the code opening that many files to blame, it seems better for commandserver to work in this case. There are multiple way to "solve" it, like preserving a fd with a small number and swap it with sock using dup2(). But upgrading to a modern selector supported by the system seems to be the most correct way.

selector2: vendor selector2 library This library was a backport of the Python 3 "selectors" library. It is useful to provide a better selector interface for Python2, to address some issues of the plain old select.select, mentioned in the next patch. The code [1] was ported using the MIT license, with some minor modifications to make our test happy: 1. "# no-check-code" was added since it's foreign code. 2. "from __future__ import absolute_import" was added. 3. "from collections import namedtuple, Mapping" changed to avoid direct symbol import. [1]: https://github.com/SethMichaelLarson/selectors2/blob/d27dbd2fdc48331fb76ed431f44b6e6956de7f82/selectors2.py # no-check-commit

context: name files relative to cwd in warning messages I was several directories deep in the kernel tree, ran `hg add`, and got the warning about the size of one of the files. I noticed that it suggested undoing the add with a specific revert command. The problem is, it would have failed since the path printed was relative to the repo root instead of cwd. While here, I just fixed the other messages too. As an added benefit, these messages now look the same as the verbose/inexact messages for the corresponding command. I don't think most of these messages are reachable (typically the corresponding cmdutil function does the check). I wasn't able to figure out why the keyword tests were failing when using pathto()- I couldn't cause an absolute path to be used by manipulating the --cwd flag on a regular add. (I did notice that keyword is adding the file without holding wlock.)

run-tests: disable color on Windows More Windows sadness. Maybe someone can figure out how to make win32 color work, but I think we avoid importing stuff from the mercurial package in this module. On the plus side, this conditionalizes away a test failure.

codemod: register core configitems using a script This is done by a script [2] using RedBaron [1], a tool designed for doing code refactoring. All "default" values are decided by the script and are strongly consistent with the existing code. There are 2 changes done manually to fix tests: [warn] mercurial/exchange.py: experimental.bundle2-output-capture: default needs manual removal [warn] mercurial/localrepo.py: experimental.hook-track-tags: default needs manual removal Since RedBaron is not confident about how to indent things [2]. [1]: https://github.com/PyCQA/redbaron [2]: https://github.com/PyCQA/redbaron/issues/100 [3]: #!/usr/bin/env python # codemod_configitems.py - codemod tool to fill configitems # # Copyright 2017 Facebook, Inc. # # This software may be used and distributed according to the terms of the # GNU General Public License version 2 or any later version. from __future__ import absolute_import, print_function import os import sys import redbaron def readpath(path): with open(path) as f: return f.read() def writepath(path, content): with open(path, 'w') as f: f.write(content) _configmethods = {'config', 'configbool', 'configint', 'configbytes', 'configlist', 'configdate'} def extractstring(rnode): """get the string from a RedBaron string or call_argument node""" while rnode.type != 'string': rnode = rnode.value return rnode.value[1:-1] # unquote, "'str'" -> "str" def uiconfigitems(red): """match *.ui.config* pattern, yield (node, method, args, section, name)""" for node in red.find_all('atomtrailers'): entry = None try: obj = node[-3].value method = node[-2].value args = node[-1] section = args[0].value name = args[1].value if (obj in ('ui', 'self') and method in _configmethods and section.type == 'string' and name.type == 'string'): entry = (node, method, args, extractstring(section), extractstring(name)) except Exception: pass else: if entry: yield entry def coreconfigitems(red): """match coreconfigitem(...) pattern, yield (node, args, section, name)""" for node in red.find_all('atomtrailers'): entry = None try: args = node[1] section = args[0].value name = args[1].value if (node[0].value == 'coreconfigitem' and section.type == 'string' and name.type == 'string'): entry = (node, args, extractstring(section), extractstring(name)) except Exception: pass else: if entry: yield entry def registercoreconfig(cfgred, section, name, defaultrepr): """insert coreconfigitem to cfgred AST section and name are plain string, defaultrepr is a string """ # find a place to insert the "coreconfigitem" item entries = list(coreconfigitems(cfgred)) for node, args, nodesection, nodename in reversed(entries): if (nodesection, nodename) < (section, name): # insert after this entry node.insert_after( 'coreconfigitem(%r, %r,\n' ' default=%s,\n' ')' % (section, name, defaultrepr)) return def main(argv): if not argv: print('Usage: codemod_configitems.py FILES\n' 'For example, FILES could be "{hgext,mercurial}/*/**.py"') dirname = os.path.dirname reporoot = dirname(dirname(dirname(os.path.abspath(__file__)))) # register configitems to this destination cfgpath = os.path.join(reporoot, 'mercurial', 'configitems.py') cfgred = redbaron.RedBaron(readpath(cfgpath)) # state about what to do registered = set((s, n) for n, a, s, n in coreconfigitems(cfgred)) toregister = {} # {(section, name): defaultrepr} coreconfigs = set() # {(section, name)}, whether it's used in core # first loop: scan all files before taking any action for i, path in enumerate(argv): print('(%d/%d) scanning %s' % (i + 1, len(argv), path)) iscore = ('mercurial' in path) and ('hgext' not in path) red = redbaron.RedBaron(readpath(path)) # find all repo.ui.config* and ui.config* calls, and collect their # section, name and default value information. for node, method, args, section, name in uiconfigitems(red): if section == 'web': # [web] section has some weirdness, ignore them for now continue defaultrepr = None key = (section, name) if len(args) == 2: if key in registered: continue if method == 'configlist': defaultrepr = 'list' elif method == 'configbool': defaultrepr = 'False' else: defaultrepr = 'None' elif len(args) >= 3 and (args[2].target is None or args[2].target.value == 'default'): # try to understand the "default" value dnode = args[2].value if dnode.type == 'name': if dnode.value in {'None', 'True', 'False'}: defaultrepr = dnode.value elif dnode.type == 'string': defaultrepr = repr(dnode.value[1:-1]) elif dnode.type in ('int', 'float'): defaultrepr = dnode.value # inconsistent default if key in toregister and toregister[key] != defaultrepr: defaultrepr = None # interesting to rewrite if key not in registered: if defaultrepr is None: print('[note] %s: %s.%s: unsupported default' % (path, section, name)) registered.add(key) # skip checking it again else: toregister[key] = defaultrepr if iscore: coreconfigs.add(key) # second loop: rewrite files given "toregister" result for path in argv: # reconstruct redbaron - trade CPU for memory red = redbaron.RedBaron(readpath(path)) changed = False for node, method, args, section, name in uiconfigitems(red): key = (section, name) defaultrepr = toregister.get(key) if defaultrepr is None or key not in coreconfigs: continue if len(args) >= 3 and (args[2].target is None or args[2].target.value == 'default'): try: del args[2] changed = True except Exception: # redbaron fails to do the rewrite due to indentation # see https://github.com/PyCQA/redbaron/issues/100 print('[warn] %s: %s.%s: default needs manual removal' % (path, section, name)) if key not in registered: print('registering %s.%s' % (section, name)) registercoreconfig(cfgred, section, name, defaultrepr) registered.add(key) if changed: print('updating %s' % path) writepath(path, red.dumps()) if toregister: print('updating configitems.py') writepath(cfgpath, cfgred.dumps()) if __name__ == "__main__": sys.exit(main(sys.argv[1:]))

phabricator: allow specifying reviewers on phabsend Sometimes people want to specify reviewer explicitly for a stack. The webpage only allows changing reviewer for one revision at a time. This patch adds a `--reviewer` flag to make it easier to specify reviewers. Test Plan: On a test Phabricator instance, enable `differential.allow-self-accept`, assign myself as a reviewer and make sure it works. Also try an invalid username and make sure it raises. Differential Revision: https://phab.mercurial-scm.org/D38

match: remove unused negatematcher This was only used by the sparse extension's dirstate._ignore override, which no longer exists. Differential Revision: https://phab.mercurial-scm.org/D60

sparse: override dirstate.walk() instead of dirstate._ignore Instead of treating files that are outside the sparse config as ignored, this makes it so we list only those that are within the sparse config by passing the sparse matcher to dirstate.walk(). Once we add support for narrow (sparseness applied to history, not just working copy), we will need to do a similar restriction of the walk over manifests, so this will be more consistent then. It also simplifies the code a bit. Note that a side-effect of this change is that files outside the sparse config used to be listed as ignored, but they will now not be listed at all. This can be seen in the test case where "hg purge" no longer has any effect because it doesn't see that the files outside the space config exist. To fix that, I think we should add an option to dirstate.walk() to walk outside the sparse config. We might expose that to the user as --no-sparse flag to e.g. "hg status" and "hg purge", but that's work for another day. Differential Revision: https://phab.mercurial-scm.org/D59

patch: use devel.all-warnings to replace devel.all It appears to be a misspell in patch.py.

sslutil: inform the user about how to fix an incomplete certificate chain This is a Windows only thing. Unfortunately, the socket is closed at this point (so the certificate is unavailable to check the chain). That means it's printed out when verification fails as a guess, on the assumption that 1) most of the time verification won't fail, and 2) sites using expired or certs that are too new will be rare. Maybe this is an argument for adding more functionality to debugssl, to test for problems and print certificate info. Or maybe it's an argument for bundling certificates with the Windows builds. That idea was set aside when the enhanced SSL code went in last summer, and it looks like there were issues with using certifi on Windows anyway[1]. This was tested by deleting the certificate out of certmgr.msc > "Third-Party Root Certification Authorities" > "Certificates", seeing `hg pull` fail (with the new message), trying this command, and then successfully performing the pull command. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/089573.html

debug: add a method to check the state of, and built an SSL cert chain This is only useful on Windows, and avoids the need to use Internet Explorer to build the certificate chain. I can see this being extended in the future to print information about the certificate(s) to help debug issues on any platform. Maybe even perform some of the python checks listed on the secure connections wiki page. But for now, all I need is 1) a command that can be invoked in a setup script to ensure the certificate is installed, and 2) a command that the user can run if/when a certificate changes in the future. It would have been nice to leverage the sslutil library to pick up host specific settings, but attempting to use sslutil.wrapsocket() failed the 'not sslsocket.cipher()' check in it and aborted. The output is a little more chatty than some commands, but I've seen the update take 10+ seconds, and this is only a debug command.

win32: add a method to trigger the Crypto API to complete a certificate chain I started a thread[1] on the mailing list awhile ago, but the short version is that Windows doesn't ship with a full list of certificates[2]. Even if the server sends the whole chain, if Windows doesn't have the appropriate certificate pre-installed in its "Third-Party Root Certification Authorities" store, connections mysteriously fail with: abort: error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661) Windows expects the application to call the methods invoked here as part of the certificate verification, triggering a call out to Windows update if necessary, to complete the trust chain. The python bug to add this support[3] hasn't had any recent activity, and isn't targeting py27 anyway. The only work around that I could find (besides figuring out the certificate and walking through the import wizard) is to browse to the site in Internet Explorer. Opening the page with FireFox or Chrome didn't work. That's a pretty obscure way to fix a pretty obscure problem. We go to great lengths to demystify various SSL errors, but this case is clearly lacking. Let's try to make things easier to diagnose and fix. When I had trouble figuring out how to get ctypes to work with all of the API pointers, I found that there are other python projects[4] using this API to achieve the same thing. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-April/096501.html [2] https://support.microsoft.com/en-us/help/931125/how-to-get-a-root-certificate-update-for-windows [3] https://bugs.python.org/issue20916 [4] https://github.com/nvaccess/nvda/blob/3b86bce2066b1934df14b96f2e83369900860ecf/source/updateCheck.py#L511

bookmarks: use 'applychanges' for bookmark update There is still some use of 'deletedivergent' bookmark here. They will be taken care of later. The 'deletedivergent' code needs some rework before fitting in the new world.

bookmark: use 'applychanges' in 'repair.strip'

bookmark: use 'applychanges' in the mq extension

bookmark: use 'applychanges' when stripping

bookmark: use 'applychanges' in the convert extension

bookmark: use 'applychanges' when updating bookmark in histedit

bookmark: use 'applychanges' when updating a bookmark through pushkey

bookmark: use 'applychanges' when updating from a remote

bookmark: use 'applychanges' for adding new bookmark

bookmark: use 'applychanges' for bookmark renaming

bookmark: use 'applychanges' for bookmark deletion

bookmark: introduce a 'applychanges' function to gather bookmark movement We want to track bookmark movement within a transaction. For this we need a more centralized way to update bookmarks. For this purpose we introduce a new 'applychanges' method that apply a list of changes encoded as '(name, node)'. We'll cover all bookmark updating code to this new method in later changesets and add bookmark move in the transaction when all will be migrated.

obsstore: keep self._data updated with _addmarkers This makes sure obsstore._data is still correct with added markers. The '_data' propertycache was added in 17ce57b7873f.

match: make base matcher return True for visitdir If a matcher doesn't implement visitdir, we should be returning True so that tree traversals are not prematurely pruned. The old value of False would prevent tree traversals when using any matcher that didn't implement visitdir. Differential Revision: https://phab.mercurial-scm.org/D83

tests: fix an incorrect description in test-ignore.t Differential Revision: https://phab.mercurial-scm.org/D82

templatekw: hide {peerpaths} keyword for 4.3 Thinking a bit further about list/dict subscript operation (proposed by issue 5534), I noticed the current data structure, a dict of dicts, might not be ideal. For example, if there were "'[' index ']'" and "'.' key" operators, "{parents[0]}" would return "{p1rev}:{p1node}", and we would probably want to write "{parents[0].desc}" to get the first element of "{parents % "{desc}"}". This will basically execute parents[0].makemap()['desc'] in Python. Given the rule above, "{peerpaths.default.pushurl}" will be translated to peerpaths['default'].makemap()['pushurl'], which means {peerpaths} should be a single-level dict and sub-options should be makemap()-ed. "{peerpaths % "{name} = {url}, {pushurl}, ..."}" (Well, it could be peerpaths['default']['pushurl'], but in which case, peerpaths['default'] should be a plain dict, not a hybrid object.) So, let's mark the current implementation experimental and revisit it later.

parsers: fix invariant bug in find_deepest (issue5623) find_deepest is used to find the "best" ancestors given a list. In the main loop it keeps an invariant called 'ninteresting' which is supposed to contain the number of non-zero entries in the 'interesting' array. This invariant is incorrectly maintained, however, which leads the the algorithm returning an empty result for certain graphs. This has been fixed. Also, the 'interesting' array is supposed to fit 2^ancestors values, but is incorrectly allocated to twice that size. This has been fixed as well. The tests in test-ancestor.py compare the Python and C versions of the code, and report the error correctly, since the Python version works correct. Even so, I have added an additional test against the expected result, in the event that both algorithms have an identical error in the future. This fixes issue5623.

configitems: register the 'worker.backgroundclose' config

configitems: register the 'progress.width' config

configitems: register the 'color.pagermode' config

configitems: handle case were the default value is not static In some case, the default of one value is derived from other value. We add a way to register them anyway and an associated devel-warning. The registration is very naive for the moment. We might be able to have a better way for registering each of these cases but it could be done later.

bugzilla: move the default regexp for fix in the config declaration This mimic the change requested by Yuya for '_default_bug_re'.

configitems: register the 'bugzilla.version' config

configitems: register the 'bugzilla.usermap' config

configitems: register the 'bugzilla.user' config

configitems: register the 'bugzilla.timeout' config

configitems: register the 'bugzilla.template' config

configitems: register the 'bugzilla.style' config

configitems: register the 'bugzilla.strip' config

configitems: register the 'bugzilla.regexp' config The default value is moved from the class to the config registration.

changegroup: stop returning and recording added nodes in 'cg.apply' cg.apply used to returns the added nodes. Callers doesn't have a use for it anymore, remove the added node and stops recording it in the current operation. This information was added in the current release cycle so no extensions breakage should happens.

phases: remove trace of addednodes in the 'phase-heads' handling updatephases have no use of the 'addednodes' parameter since 50243c975fc2. However caller are still passing it for nothing, remove the parameter and remove computing of the added nodes in caller.

phases: test phases tracking at the transaction level Now that we have all tracking in place, the data in `tr.changes['phases']` dictionary should be correct and we should test it. It is a bit late in the cycle to discuss to add any public API (eg: hooks) that expose the data to the user, so we just add a small test extension displaying the data. It is enabled for the phases tests. New output have been manually checked for consistency.

phases: track phase changes from 'retractboundary' We adds new computation to find and record the revision affected by the boundary retraction. This add more complication to the function but this seems fine since it is only used in a couple of rare and explicit cases (`hg phase --force` and `hg qimport`). Having strong tracking of phase changes is worth the effort.

phases: detect when boundaries has been actually retracted It is useful to detect noop and avoid expensive operations in this case. We return the information to inform the caller of a possible update. Top level function might need to react to the phase update (eg: invalidating some caches, tracking phase change).

phases: rework phase movement code in 'cg.apply' to use 'registernew' We rework the code to call 'registernew' before any other phase advancement. This make 'changegroup.apply' register correct phase movement for the added and bundled nodes.

convert: use the new 'phase.registernew' function

localrepo: use the 'registernew' function to set the phase of new commit

phases: add a 'registernew' method to set new phases This new function will be used by code that adds new changesets. It ajusts the phase boundary to make sure added changesets are at least in their target phase (they end up in an higher phase if their parents are in a higher phase). Having a dedicated function also simplify the phases tracking. All the new nodes are passed as argument, so we know that all of them needs to have their new phase registered. We also know that no other nodes will be affected, so no extra computation are needed. This function differ from 'retractboundary' where some nodes might change phase while some other might not. It can also affect nodes not passed as parameters. These simplification also apply to the computation itself. For now we use '_retractboundary' there by convenience, but we may introduces simpler code later. While registering new revisions, we still need to check the actual phases of the added node because it might be higher than the target phase (eg: target is draft but parent is secret). We will migrate users over the next changesets.

phases: extract the core of boundary retraction in '_retractboundary' At the moment the 'retractboundary' function is called for multiple reasons: First, actually retracting boundaries. There are only two cases for theses: 'hg phase --force' and 'hg qimport'. This will need extra graph computation to retrieve the phase changes. Second, setting the phases of newly added changesets. In this case we already know all the affected nodes and we just needs to register different information (old phase is None). Third, when reducing the set of roots when advancing phase. The phase are already properly tracked so we do not needs anything else in this case. To deal with this difference in phase tracking, we extract the core logic into a private method that all three cases can use.

phases: track phase movements in 'advanceboundary' Makes advanceboundary record the phase movement of affected revisions in tr.changes['phases']. The tracking is not usable yet because the 'retractboundary' function can also affect phases. We'll improve that in the coming changesets.

phases: extract the intermediate set of affected revs When advancing phases, we compute the new roots for the phases above. During this process, we need to compute all the revisions that change phases (to the new target phases). Extract these revisions into a separate variable. This will be useful to record the phase changes in the transaction.

phase: put retractboundary out of the loop in advanceboundary It seems that we were calling retractboundary for each phases to process. Putting the retractboundary out of the loop reduce the number of calls, helping tracking the phases changes.

match: make unionmatcher a proper matcher unionmatcher is currently used where only a limited subset of its functions will be called. Specifically, visitdir() is never called. The next patch will pass it to dirstate.walk() where it will matter that visitdir() is correctly implemented, so let's fix that. Also add the explicitdir etc that will also be assumed by dirstate.walk() to exist on a matcher. Differential Revision: https://phab.mercurial-scm.org/D58

match: write forceincludematcher using unionmatcher The forceincludematcher is simply a unionmatcher of a includematcher (matching paths recursively) with the given matcher. Since the forceincludematcher is only used by sparse, move it there. I don't have a good sparse repo setup to test performance impact on. Differential Revision: https://phab.mercurial-scm.org/D57

histedit: extract InterventionRequired transaction handling to utils rebase will have similar logic, so let's extract it. Besides, it makes the histedit code more readable. We may want to parametrize acceptintervention() by the exception(s) that should result in transaction close. Differential Revision: https://phab.mercurial-scm.org/D66

histedit: create transaction outside of try Just a little refactoring to simplify the next patch. Differential Revision: https://phab.mercurial-scm.org/D65

histedit: remove transaction from state object The transaction is only used within a single function, so no need to store it on the state object. Differential Revision: https://phab.mercurial-scm.org/D64

phabricator: verify local tags before trusting them Previously we trust local tags blindly and that could cause wrong Differential Revision to be updated, when people switch between Phabricator instances. This patch adds verification logic to detect such issue and remove problematic tags. For example, a tag "D19" was on node "X", the code will fetch all diffs attached to D19, and check if nodes server-side overlaps with nodes in precursors. If they do not overlap, create a new Differential Revision. Test Plan: Use a test Phabricator instance, send patches using `hg phabsend`, then change the local tag manually to a wrong Differential Revision number. Amend the patch and send again. Make sure the tag gets ignored and deleted. Differential Revision: https://phab.mercurial-scm.org/D36

phabricator: finding old nodes in batch This allows us to do extra sanity checks using batch APIs to prevent updating a wrong revision, which could happen when people switch Phabricator instances and having stale tags living in the repo. Differential Revision: https://phab.mercurial-scm.org/D34

phabricator: respect metadata sent by arc Previously we only respect hg:meta sent by phabsend. This patch makes it respect local:commits sent by arc as well. This avoids issues where phabread could lose the author information. Test Plan: Commit using a customized user, send the patch using arc to a test Phabricator instance, and then read the patch using phabread. Make sure it preserves the user information. Differential Revision: https://phab.mercurial-scm.org/D33

dirstate: update backup functions to take full backup filename Update the dirstate functions so that the caller supplies the full backup filename rather than just a prefix and suffix. The localrepo code was already hard-coding the fact that the backup name must be (exactly prefix + "dirstate" + suffix): it relied on this in _journalfiles() and undofiles(). Making the caller responsible for specifying the full backup name removes the need for the localrepo code to assume that dirstate._filename is always "dirstate". Differential Revision: https://phab.mercurial-scm.org/D68

util: remove unused ctxmanager This was meant as a substitute for Python's "with" with multiple context managers before we moved to Python 2.7. We're now on 2.7, so we should have no reason to keep ctxmanager. "hg grep --all ctxmanager" says that it was never used anyway. Differential Revision: https://phab.mercurial-scm.org/D73

codemod: simplify nested withs This is the result of running: python codemod_nestedwith.py **/*.py where codemod_nestedwith.py looks like this: #!/usr/bin/env python # codemod_nestedwith.py - codemod tool to rewrite nested with # # Copyright 2017 Facebook, Inc. # # This software may be used and distributed according to the terms of the # GNU General Public License version 2 or any later version. from __future__ import absolute_import, print_function import sys import redbaron def readpath(path): with open(path) as f: return f.read() def writepath(path, content): with open(path, 'w') as f: f.write(content) def main(argv): if not argv: print('Usage: codemod_nestedwith.py FILES') for i, path in enumerate(argv): print('(%d/%d) scanning %s' % (i + 1, len(argv), path)) changed = False red = redbaron.RedBaron(readpath(path)) processed = set() for node in red.find_all('with'): if node in processed or node.type != 'with': continue top = node child = top[0] while True: if len(top) > 1 or child.type != 'with': break # estimate line length after merging two "with"s new = '%swith %s:' % (top.indentation, top.contexts.dumps()) new += ', %s' % child.contexts.dumps() # only do the rewrite if the end result is within 80 chars if len(new) > 80: break processed.add(child) top.contexts.extend(child.contexts) top.value = child.value top.value.decrease_indentation(4) child = child[0] changed = True if changed: print('updating %s' % path) writepath(path, red.dumps()) if __name__ == "__main__": sys.exit(main(sys.argv[1:])) Differential Revision: https://phab.mercurial-scm.org/D77

reposvfs: add a ward to check if locks are properly taken we wrap 'repo.svfs.audit' to check for the store lock when accessing file in '.hg/store' for writing. This caught a couple of instance where the transaction was released after the lock, we should probably have a dedicated checker for that case.

repovfs: add a ward to check if locks are properly taken When the appropriate developer warnings are enabled, We wrap 'repo.vfs.audit' to check for locks when accessing file in '.hg' for writing. Another changeset will add a 'ward' for the store vfs (svfs). This check system has caught a handful of locking issues that have been fixed in previous series (mostly in 4.0). I expect another batch to be caught in third party extensions. We introduce two real exceptions from extensions 'blackbox.log' (because a lot of read-only operations add entry to it), and 'last-email.txt' (because 'hg email' is currently a read only operation and there is value to keep it this way). In addition we are currently allowing bisect to operate outside of the lock because the current code is a bit hard to get properly locked for now. Multiple clean up have been made but there is still a couple of them to do and the freeze is coming.

vfs: allow to pass more argument to audit We want to be able to do more precise check when auditing a path depending of the intend of the file access (eg read versus write). So we now pass the 'mode' value to 'audit' and update the audit function to accept them. This will be put to use in the next changeset.

zsh_completion: add '--partial' flag to completions for 'import' (issue5618) When using the 'import' command running ZSH together with its Mercurial-specific completions the flag '--partial' (introduced in Mercurial 3.1, 2014-08-01) is not offered as completion option. This patch adds it to the list of completions for the 'import' command.

configitems: register the 'bugzilla.password' config

configitems: register the 'bugzilla.mocklog' config

test-amend: fix HGEDITOR shell script to run on Windows Windows doesn't know how to launch *.sh directly. This workaround is used in several other tests.

test-paths: mask out JSON specific path differences for Windows The backslashes in the local paths were being escaped with another backslash, and $TESTTMP doesn't match against the double backslashed path. This doesn't happen without the 'json' filter.

tests: categorize entries in blacklist for fsmonitor for future reference

tests: avoid tests related to directory permission if running with fsmonitor

tests: add extra output for fsmonitor at checking under .hg

tests: add extra output for fsmonitor at showconfig

tests: take extra care for fsmonitor at enabling incompatible extension This is worthwhile, because these tests don't focus mainly on using incompatible extensions itself.

tests: remove meaningless entries from blacklist for fsmonitor IMHO, these tests were listed up in blacklist at 3b67f27bb908, because fsmonitor extension wasn't correctly installed before actual testing at that revision (this issue was fixed by 776fd2e2cf5a).

tests: remove meaningless comment from blacklist for fsmonitor These tests can run successfully since fe0667cc521e.

test-https: stabilize for Windows A leading '/' without a following drive letter is resolved to the location of the MSYS installation. These could be collapsed into a single line of output, but this seems more self documenting.

tagmerge: use workingfilectx to write merged tags This function already does an excellent job of reading from context objects; we simply need to change the single write call to eliminate all uses of the wvfs. As with past changes, the effect should be a no-op but opens the door to in-memory merge later by using different context objects.

run-tests: add color to output if pygments is available The output of run-tests has no formatting by default, which hampers readability. This patch colors the diff output when pygments is available. To avoid coloring even when pygments is available, use --color never.

win32: work around a WinError problem handling HRESULT types I ran into this ctypes bug while working with the Crypto API. While this could be an issue with any Win32 API in theory, the handful of things that we call are older functions that are unlikely to return COM errors, so I didn't retrofit this everywhere.

test-check-pyflakes: do not leave test.py in the repo Differential Revision: https://phab.mercurial-scm.org/D67

revset: add experimental ancestors/descendants relation subscript The relation name is 'generations' now, which may be changed in future.

revset: add experimental relation and subscript operators The proposed syntax [1] was originally 'set{n rel}', but it seemed slightly confusing if template is involved. On the other hand, we want to keep 'set[n]' for future extension. So this patch introduces 'set#rel[n]' ternary operator. I chose '#' just because it looks like applying an attribute. This also adds stubs for 'set[n]' and 'set#rel' operators since these syntax elements are fundamental for constructing 'set#rel[n]'. [1]: https://www.mercurial-scm.org/wiki/RevsetOperatorPlan#ideas_from_mpm

revset: do not compute weight for integer literal argument In x^n and x~n, n isn't a set expression. There's no need to optimize the right-hand side.

templatekw: export ui.paths as {peerpaths} It's sometimes useful to show hyperlinks in log output. "{get(peerpaths, "default")}/rev/{node}" Since each path may have sub options, "{peerpaths}" is structured as a dict of dicts, but the inner dict is rendered as if it were a string URL. The implementation is ad-hoc, so there are some weird behaviors described in the test. We might need to introduce a proper way of handling a hybrid scalar object. This patch adds _hybrid.__getitem__() so d['path']['url'] works. The keyword is named as "peerpaths" since "paths" seemed too generic in log context.

summary: fix type of empty unresolved list It was okay because tested as a boolean prior to calling len(), but looked incorrect.

vfs: rename auditvfs to proxyvfs Since we've removed mustaudit property, auditvfs has no auditing business. It's just a utility class for vfs wrappers.

streamclone: comment why path auditing is disabled in generatev1() Copied from 39c6e349dfff. I wasn't sure whether it's for optimization or suppressing unwanted error.

streamclone: close large revlog files explicitly in generatev1()

bundle2: no longer use 'retractboundary' in updatephases The new 'phase-heads' forced all added node to secret before advancing the boundary to work around the fact changesets were added as draft by default. This is no longer necessary since the changegroup part can now use the 'targetphase' parameter. Not doing this retract boundary call has a couple of advantages: * This makes implementing phases change tracking in the transaction much simpler since retract boundary can become a rare case. * Bundling secret changesets is not the norm. Exchange never does that and even for strip, the use-case is not common.Skipping the retract boundary will avoid useless work here. * Sending phase update on push can be simplified since we can rely on the behavior of 'cg.apply' for most of it. This means less phases update send for example. * We no longer needs to track and use the addednodes during unbundling. This make it possible to have multiple 'changegroup' and 'phase-heads' parts in the same bundle without them interfering with each others. The new part has not been part of any release yet so we do not offer backward compatibility yet. It is important to update this semantic before the 4.3 freeze happens.

bundle2: automatically add 'targetphase' parameter in writenewbundle If we are bundling secret changeset and the bundle will contain phase, we request the changegroup to be applied as secret. It will be useful for next patch as we are now sure that secrets changesets are applied as secret and not applied as draft then forced to secret.

bundle2: support the 'targetphase' parameter for the changegroup part By default unbundled changesets are drafts. We want to reduce the number of phases changes during unbundling by giving the possibility to the bundle to indicate the phase of unbundled changesets. The longer terms goal is to add phase movement tracking in tr.changes and the 'retractboundary' call is making it more complicated than we want.

changegroup: stop treating strip as special when dealing with phases Since 8e3021fd1a44, the strip bundle includes the phases of the stripping node. Hence we don't need this special case anymore. Dropping it will helps make the phase behavior more consistent across all exchanges medium.

match: inverse _anypats(), making it _prefix()

amend: new extension providing the amend command Various third parties have implemented the `amend` command, which is in high demand. This patch adds it as an experimental extension so its interface could be formalized in core directly. Since `commit --amend` is basically what `amend` should do. The command is just a thin wrapper around `commit --amend` and just prevent the editor from popping up by passing `--message`.

py3: make localrepo filtered repo cache work on py3 I don't know if this is the right fix, but it makes test-py3-commands.t pass again. Differential Revision: https://phab.mercurial-scm.org/D56

configitems: register the 'bugzilla.host' config

configitems: register the 'bugzilla.fixstatus' config

configitems: register the 'bugzilla.fixresolution' config

configitems: register the 'bugzilla.fixregexp' config

configitems: register the 'bugzilla.db' config

configitems: register the 'bugzilla.bzuser' config

configitems: register the 'bugzilla.bzurl' config

configitems: register the 'bugzilla.bzemail' config

configitems: register the 'bugzilla.bzdir' config

configitems: register the 'bugzilla.apikey' config

revlog: use struct.Struct instances for slight performance wins Differential Revision: https://phab.mercurial-scm.org/D32

revlog: micro-optimize the computation of hashes Differential Revision: https://phab.mercurial-scm.org/D31

hgweb: re-implement followlines UI selection using buttons This changeset attempts to solve two issues with the "followlines" UI in hgweb. First the "followlines" action is currently not easily discoverable (one has to hover on a line for some time, wait for the invite message to appear and then perform some action). Second, it gets in the way of natural line selection, especially in filerevision view. This changeset introduces an additional markup element (a <button class="btn-followlines">) alongside each content line of the view. This button now holds events for line selection that were previously plugged onto content lines directly. Consequently, there's no more action on content lines, hence restoring the "natural line selection" behavior (solving the second problem). These buttons are hidden by default and get displayed upon hover of content lines; then upon hover of a button itself, a text inviting followlines section shows up. This solves the first problem (discoverability) as we now have a clear visual element indicating that "some action could be perform" (i.e. a button) and that is self-documented. In followlines.js, all event listeners are now attached to these <button> elements. The custom "floating tooltip" element is dropped as <button> elements are now self-documented through a "title" attribute that changes depending on preceding actions (selection started or not, in particular). The new <button> element is inserted in followlines.js script (thus only visible if JavaScript is activated); it contains a "+" and "-" with a "diff-semantics" style; upon hover, it scales up. To find the parent element under which to insert the <button> we either rely on the "data-selectabletag" attribute (which defines the HTML tag of children of class="sourcelines" element e.g. <span> for filerevision view and <tr> for annotate view) or use a child of the latter elements if we find an element with class="followlines-btn-parent" (useful for annotate view, for which we have to find the <td> in which to insert the <button>). On noticeable change in CSS concerns the "margin-left" of span:before pseudo-elements in filelog view that has been increased a bit in order to leave space for the new button to appear between line number column and line content one. Also note the "z-index" addition for "annotate-info" box so that the latter appears on top of new buttons (instead of getting hidden). In some respect, the UI similar to line commenting feature that is implemented in popular code hosting site like GitHub, BitBucket or Kallithea.

localrepo: cache types for filtered repos (issue5043) Python introduces a reference cycle on dynamically created types via __mro__, making them very easy to leak. See https://bugs.python.org/issue17950. Previously, repo.filtered() created a type on every invocation. Long-running processes (like `hg convert`) could call this function thousands of times, leading to a steady memory leak. Since we're Unable to stop the leak because this is a bug in Python, the next best thing is to contain it. This patch adds a cache of of the dynamically generated repoview/filter types on the localrepo object. Since we only generate each type once, we cap the amount of memory that can leak to something reasonable. After this change, `hg convert` no longer leaks memory on every revision. The process will likely grow memory usage over time due to e.g. larger manifests. But there are no leaks.

convert: transcode CVS log messages by specified encoding (issue5597) Converting from CVS to Mercurial assumes that CVS log messages in "cvs rlog" output are encoded in UTF-8 (or basic Latin-1). But cvs itself is usually unaware of encoding of log messages, in practice. Therefore, if there are commits, of which log message is encoded in other than UTF-8, log message of corresponded revisions in the converted repository will be broken. To avoid such broken log messages, this patch transcodes CVS log messages by encoding specified via "convert.cvsps.logencoding" configuration. This patch accepts multiple encoding for convenience, because "multiple encoding mixed in a repository" easily occurs. For example, UTF-8 (recent POSIX), cp932 (Windows), and EUC-JP (legacy POSIX) are well known encoding for Japanese.

fsmonitor: execute setup procedures only if dirstate is already instantiated Before this patch, reposetup() of fsmonitor executes setup procedures for dirstate, even if it isn't yet instantiated at that time. On the other hand, dirstate might be already instantiated before reposetup() intentionally (prefilling by chg, for example, see bf3af0eced44 for detail). If so, just discarding already instantiated one in reposetup() causes issue. To resolve both issues above, this patch executes setup procedures, only if dirstate is already instantiated. BTW, this patch removes "del repo.unfiltered().__dict__['dirstate']", because it is responsibility of the code path, which causes instantiation of dirstate before reposetup(). After this patch, using localrepo.isfilecached() should avoid creating the corresponded entry in repo.unfiltered().__dict__.

fsmonitor: centralize setup procedures for dirstate

fsmonitor: avoid needless instantiation of dirstate Using repo.local() instead of util.safehasattr(repo, 'dirstate') also avoids executing setup procedures for remote repository (including statichttprepo). This is reason why this patch also removes a part of subsequent comment, and try/except for AttributeError at accessing to repo.wvfs.

journal: use wrapfilecache instead of wrapfunction on func of filecache wrapfilecache() on filecache-ed property works more strictly than wrapfunction() directly on func() of filecache.

journal: execute setup procedures for already instantiated dirstate If dirstate is instantiated before reposetup() of journal extension, it doesn't have "journalstorage" property, even if it is instantiated via wrapdirstate() wrapping repo.dirstate(), because wrapdirstate() works as same as original one before marking repo as "journal"-ing in reposetup(). This issue can be reproduced by running test-journal.t or test-journal-share.t with fsmonitor-run-tests.py. On the other hand, just discarding already instantiated dirstate in reposetup() prevents chg from filling dirstate before reposetup() (see bf3af0eced44 for detail). Therefore, this patch executes setup procedures for already instantiated dirstate explicitly in reposetup(). To centralize setup procedures for dirstate, this patch also factors them out from wrapdirstate().

localrepo: add isfilecached to check filecache-ed property is already cached isfilecached() encapsulates internal implementation of filecache-ed property. "name in repo.unfiltered().__dict__" or so can't be used for this purpose, because corresponded entry in __dict__ might be discarded by repo.invalidate(), repo.invalidatedirstate() or so (fsmonitor does so, for example). This patch makes isfilecached() return not only whether filecache-ed property is already cached, but also already cached value (or None), in order to avoid subsequent access to cached object via "repo.NAME", which prevents main Mercurial procedure after reposetup() from validating cache.

sslutil: check for missing certificate and key files (issue5598) Currently, sslutil._hostsettings() performs validation that web.cacerts exists. However, client certificates are passed in to the function and not all callers may validate them. This includes httpconnection.readauthforuri(), which loads the [auth] section. If a missing file is specified, the ssl module will raise a generic IOException. And, it doesn't even give us the courtesy of telling us which file is missing! Mercurial then prints a generic "abort: No such file or directory" (or similar) error, leaving users to scratch their head as to what file is missing. This commit introduces explicit validation of all paths passed as arguments to wrapsocket() and wrapserversocket(). Any missing file is alerted about explicitly. We should probably catch missing files earlier - as part of loading the [auth] section. However, I think the sslutil functions should check for file presence regardless of what callers do because that's the only way to be sure that missing files are always detected.

match: override matchfn instead of __call__ for consistency The matchers that were recently moved into core from the sparse extension override __call__, while the previously existing matchers override matchfn. Let's switch to the latter for consistency.

match: express anypats(), not prefix(), in terms of the others When I added prefix() in 9789b4a7c595 (match: introduce boolean prefix() method, 2014-10-28), we already had always(), isexact(), and anypats(), so it made sense to write it in terms of them (a prefix matcher is one that isn't any of the other types). It's only now that I realize that it's much more natural to define prefix() explicitly (it's one that uses path: patterns, roughly speaking) and let anypats() be defined in terms of the others. Remember that these methods are all used for determining which fast paths are possible. anypats() simply means that no fast paths are possible (it could be called complex() instead). Further evidence is that rootfilesin:some/dir does not have any patterns, but it's still considered to be an anypats() matcher. That's because anypats() really just means that it's not a prefix() matcher (and not always() and not isexact()). This patch thus changes prefix() to return False by default and anypats() to return True only if the other three are False. Having anypats() be True by default also seems like a good thing, because it means forgetting to override it will lead only to performance bugs, not correctness bugs. Since the base class's implementation changes, we're also forced to update the subclasses. That change exposed and fixed a bug in the differencematcher: for example when both its two input matchers were prefix matchers, we would say that the result was also a prefix matcher, which is incorrect, because e.g "path:dir - path:dir/foo" no longer matches everything under "dir" (which is what prefix() means).

match: make nevermatcher an exact matcher and a prefix matcher The m.isexact() and m.prefix() methods are used by callers to determine whether m.files() can be used for fast paths. It seems safe to let callers to any fast paths it can that rely on the empty m.files().

revset: define successors revset This revset returns all successors, including transit nodes and the source nodes (to be consistent with existing revsets like "ancestors"). To filter out transit nodes, use `successors(X)-obsolete()`. To filter out divergent case, use `successors(X)-divergent()-obsolete()`. The revset could be useful to define rebase destination, like: `max(successors(BASE)-divergent()-obsolete())`. The `max` is to deal with splits. There are other implementations where `successors` returns just one level of successors, and `allsuccessors` returns everything. I think `successors` returning all successors by default is more user friendly. We have seen cases in production where people use 1-level `successors` while they really want `allsuccessors`. So it seems better to just have one single revset returning all successors by default to avoid user errors. In the future we might want to add `depth` keyword argument to it and for other revsets like `ancestors` etc. Or even build some flexible indexing syntax [1] to satisfy people having the depth limit requirement. [1]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-July/101140.html

sparse: shorten try..except block in updateconfig() It now only covers refreshwdir(). This is what importfromfiles() does. I think it is the more appropriate behavior.

sparse: clean up updateconfig() * Use context manager for wlock * Rename oldsparsematch to oldmatcher * Always call parseconfig() because parsing an empty string yields the same result as the old code

sparse: move config updating function into core As part of the move, the ui argument was dropped. Additional fixups will be made in a follow-up commit.

dirstate: expose a sparse matcher on dirstate (API) The sparse extension performs a lot of monkeypatching of dirstate to make it sparse aware. Essentially, various operations need to take the active sparse config into account. They do this by obtaining a matcher representing the sparse config and filtering paths through it. The monkeypatching is done by stuffing a reference to a repo on dirstate and calling sparse.matcher() (which takes a repo instance) during each function call. The reason this function takes a repo instance is because resolving the sparse config may require resolving file contents from filelogs, and that requires a repo. (If the current sparse config references "profile" files, the contents of those files from the dirstate's parent revisions is resolved.) I seem to recall people having strong opinions that the dirstate object not have a reference to a repo. So copying what the sparse extension does probably won't fly in core. Plus, the dirstate modifications shouldn't require a full repo: they only need a matcher. So there's no good reason to stuff a reference to the repo in dirstate. This commit exposes a sparse matcher to dirstate via a property that when looked up will call a function that eventually calls sparse.matcher(). The repo instance is bound in a closure, so it isn't exposed to dirstate. This approach is functionally similar to what the sparse extension does today, except it hides the repo instance from dirstate. The approach is not optimal because we have to call a proxy function and sparse.matcher() on every property lookup. There is room to cache the matcher instance in dirstate. After all, the matcher only changes if the dirstate's parents change or if the sparse config changes. It feels like we should be able to detect both events and update the matcher when this occurs. But for now we preserve the existing semantics so we can move the dirstate sparseness bits into core. Once in core, refactoring becomes a bit easier since it will be clearer how all these components interact. The sparse extension has been updated to use the new property. Because all references to the repo on dirstate have been removed, the code for setting it has been removed.

sparse: use self instead of repo.dirstate "self" here is the dirstate instance. I'm pretty confident that self and repo.dirstate will be the exact same object. So remove a dependency on repo by just looking at self.

sparse: move code for importing rules from files into core This is a pretty straightforward port. Some code cleanup was performed. But no major changes to the logic were made. I'm not a huge fan of this function because it does multiple things. I'd like to get things into core first to facilitate refactoring later. Please also note the added inline comment about the oddities of writeconfig() and the try..except to undo it. This is because of the hackiness in which the sparse matcher is obtained by various consumers, notably dirstate. We'll need a massive refactor to address this. That refactor is effectively blocked on having the sparse dirstate hacks live in core.

sparse: refactor activeprofiles into a generic function (API) activeprofiles() is a special case of a more generic function. Furthermore, that generic function is essentially already implemented inline in the sparse extension. So, refactor activeprofiles() to a generic activeconfig(). Change the only consumer of activeprofiles() to use it. And have the inline implementation in the sparse extension use it.

check-code: prohibit `if False` antipattern Differential Revision: https://phab.mercurial-scm.org/D20

convert: remove `if False` block This code has never run since its introduction on July 18th, 2007. It's time for it to go. Differential Revision: https://phab.mercurial-scm.org/D19

filterpyflakes: move self-test into test file This will avoid a false positive on an upcoming check-code rule. Differential Revision: https://phab.mercurial-scm.org/D18

test-subrepo: demonstrate a status problem when merge deletes a file At the interactive update prompt, if (c) is chosen and then followed by `hg rm`, both `status -R` and `status -S` show the file as 'R', and `files -R` shows no files (OK, because explicitly removed files aren't supposed to be listed). If `rm` follows selecting (c), then both flavors of `status` list the file as '!', and `files -R` lists the missing file. So somehow, the (d) option has followed a third path.

subrepo: make the output references to subrepositories consistent Well, mostly. The annotation on subrepo functions tacks on a parenthetical to the abort message, which seems reasonable for a generic mechanism. But now all messages consistently spell out 'subrepository', and double quote the name of the repo. I noticed the inconsistency in the change for the last commit.

subrepo: consider the parent repo dirty when a file is missing This simply passes the 'missing' argument down from the context of the parent repo, so the same rules apply. subrepo.bailifchanged() is hardcoded to care about missing files, because cmdutil.bailifchanged() is too. In the end, it looks like this addresses inconsistencies with 'archive', 'identify', blackbox logs, 'merge', and 'update --check'. I wasn't sure how to implement this in git, so that's left for someone more familiar with it.

archival: flag missing files as a dirty wdir() in the metadata file (BC) Since the identify command adds a '+' for missing files, it's reasonable that this does too. Perhaps the node field's hex value should be p1+p2 for merges?

cmdutil: simplify the dirty check in howtocontinue() This is equivalent to the previous code. But it seems to me that if the user is going to be prompted that a commit is needed, missing files should be ignored, but branch and merge changes shouldn't be.

blackbox: simplify the dirty check Same idea (and possibly incorrect behavior) as the previous commit.

identify: simplify the dirty check This is equivalent to the previous code, but it seems better to be explicit about what aspects of dirty are being ignored. Perhaps they shouldn't be, since the help text says 'followed by a "+" if the working directory has uncommitted changes'. Both merges and branch changes are committable, even if the files are unchanged. Additionally, this will make the `identify` command notice missing subrepo files, once subrepos are taught to look for missing files.

tests: tweak the subrepo dirty state tests This is a continuation of 439b4d005b4a. I overlooked that blackbox logs also have a dirty marker. Also, the `hg update --check` test was updating to a revision where the deleted file wasn't tracked, which is why status seemed to show the deleted file was restored.

match: combine regex code for path: and relpath: The regexes for path: and relpath: patterns are the same (since the paths have already been normalized at the point we create the regexes). I don't think the "if pat == '.'" will have any effect relpath: because relpath: patterns will have the root directory already normalized to '' by pathutil.canonpath() (unlike path:, for which the root gets normalized to '.' by util.normpath()).

match: remove unnecessary '^' from regexes The regexes are passed to re.match(), which matches against the beginning of the input, so the '^' doesn't do anything. Note that unrooted patterns, such as globs and regexes from .hgignore are instead achieved by adding '.*' to the expression given by the user. (That's unless the user's expression started with '^', in which case the '.*' is not added, perhaps to keep the regex cleaner?)

sparse: access status fields by name instead of deconstructing it The status tuples has had named fields for a few years now.

sparse: move printing of sparse config changes function into core As part of the port, all arguments now have default values of 0. Strings are now also given the i18n treatment.

sparse: move code for clearing rules to core This is a pretty straightforward port.

sparse: move post commit actions into core Instead of wrapping committablectx.markcommitted(), we inline the call into workingctx.markcommitted(). Per smf's review, workingctx is the proper location for this code, as committablectx is the shared base class for it and memctx. Since this code touches the working directory, it belongs in workingctx.

cleanupnode: do not use generator for node mapping The 'successors' part of the mappings used of be a tuple. This avoid issue from code consuming the generator "by mistake". For example, an extension inspecting the mapping content used to be able to iterate over the successors mapping without consequence. Since the mapping are small we do not expect any performance impact we use tuple again for this.

histedit: use scmutil.cleanupnodes (BC) This is marked as BC because the strip backup file name has changed.

histedit: unify strip backup files on success (BC) Previously we wrote two different strip backup files on success. This patch unifies them. It will make scmutil.cleanupnodes migration more smooth.

histedit: pass multiple nodes to strip (BC) Previously, histedit.cleanupnode pass root nodes one by one. Since repair.strip takes multiple nodes and can handle them just fine, pass all strip roots at once. This is BC because the number of strip backup files may change from N to 1.

histedit: remove "name" parameter from cleanupnode functions The "name" parameter is not used any longer so let's remove it.

histedit: remove "should strip" debug message The debug message was not used anywhere. Removed it to make scmutil.cleanupnodes migration easier to reason about.

histedit: move topmost bookmark movement to a separate function histedit treats topmost bookmark movement specially. The rest of the bookmark movement could be handled by scmutil.cleanupnodes. So let's move the special logic out to make the patch easier to review.

histedit: remove moving bookmarks message on verbose (BC) This is more consistent with other commands, like "commit -v" won't show bookmark movement messages. It will make migrating to scmutil.cleanupnodes easier.

test-largefiles-update: conditionalize output instead of tests

test-status-rev: conditionalize output instead of tests

test-mq: conditionalize output instead of tests

test-annotate: conditionalize output instead of tests

test-addremove: conditionalize output instead of tests

test-tools: conditionalize output instead of tests

test-rebase: conditionalize output instead of tests This should help prevent breakage like was fixed in the last patch.

tests: stabilize on Windows I'm not sure if the difference on Windows for test-sparse.t is expected or not. It looks like unless the leading '/' is followed by a drive letter, '/' is resolved to 'C:/MinGW/msys/1.0'. But both cases abort with "not under root" instead of just warning.

revset: make repo.anyrevs accept customized alias override (API) Previously repo.anyrevs only expand aliases in [revsetalias] config. This patch makes it more flexible to accept a customized dict defining aliases without having to couple with ui. revsetlang.expandaliases now has the signature (tree, aliases, warn=None) which is more consistent with templater.expandaliases. revsetlang.py is now free from "ui", which seems to be a good thing.