sparse: add a requirement when a repository uses sparse (BC) The presence of a sparse checkout can confuse legacy clients or clients without sparse enabled for reasons that should be obvious. This commit introduces a new repository requirement that tracks whether sparse is enabled. The requirement is added when a sparse config is activated and removed when the sparse config is reset. The localrepository constructor has been taught to not open repos with this requirement unless the sparse feature is enabled. It yields a more actionable error message than what you would get if the lockout were handled strictly at the requirements verification phase. Old clients that aren't sparse aware will see the generic "repository requires features unknown to this Mercurial" error, however. The new requirement has "exp" in its name to reflect the experimental nature of sparse. There's a chance that the eventual non-experimental feature won't change significantly and we could have squatted on the "sparse" requirement without ill effect. If that happens, we can teach new clients to still recognize the old name. But I suspect we'll sneak in some BC and we'll want a new requirement to convey new meaning. Differential Revision: https://phab.mercurial-scm.org/D110

sparse: consolidate common code for writing sparse config In 3 functions we were writing the sparse config and updating the working directory. In two of them we had a transaction-like process for restoring the sparse config in case of wdir update fail. Because the pattern is common, we've already made mistakes, and the complexity will increase in the near future, let's consolidate the code into a reusable function. As part of this refactor, we end up reading the "sparse" file twice when updating it. This is a bit sub-optimal. But I don't think it is worth the code complexity to pass around the variables to avoid the redundancy. Differential Revision: https://phab.mercurial-scm.org/D109

revset: pass repo when passing ui The repo instance is currently only used to provide a changeset lookup function as part of parsing revsets. I /think/ this allows node fragments to resolve. I'm not sure why we wouldn't want this to always "just work" if parsing a revset string. Plus, an upcoming commit will introduce a new consumer that needs a handle on the repo. So passing it more often will make that code work more. Passing a repo instance in all callers of revset.match* results in a bunch of test changes. Notably, branch and tags caches get populated as part of evaluating revsets. I'm not sure if this is desirable. So this patch takes the conservative approach and only passes the repo if we're passing a ui instance. Differential Revision: https://phab.mercurial-scm.org/D97

win32: copy-edit debugssl messages to match prevailing style

run-tests: make sure to check if pygments is installed before using it e80041832e introduced support to color the output of tests but used pygments without checking whether it's installed or not. That breaks test-run-tests.t for machines which don't have pygments installed. This patch conditionalize the color test in test-run-tests.t and also add a check to make sure pygments is installed before using that.

sparse: require [section] in sparse config files (BC) Previously, [include] was implicit and pattern lines before a [section] were added to includes. Because the format may change in the future and explicit behavior, well, more explicit, this commit changes the config parser to reject pattern lines that don't occur in a [section]. Differential Revision: https://phab.mercurial-scm.org/D96

sparse: use set for capturing profiles Order doesn't need to be preserved. A set is acceptable. Differential Revision: https://phab.mercurial-scm.org/D95

util: remove dead code which used to be for old python2 versions Differential Revision: https://phab.mercurial-scm.org/D107

status: add a flag to terse the output (issue4119) This adds an experimental flag -t/--terse which will terse the output. The terse flag will respect other flags which filters the output. The flag takes a string whose value can be a subsequence of "marduic" (the order does not matter here.) Ignored files are not considered while tersing unless -i flag is passed or 'i' is there is the terse flag value. The flag is experimental for testing as there may be cases which will produce strange results with the flag. We can set the terse on by default by simply passing 'u' to the cmdutil.tersestatus(). This patch also adds a test file with tests covering the new feature.

commitextras: check the format of the arguments and no internal key is used This patch adds check to make the arguments are passed as KEY=VALUE and no key which is used internally is passed. This patch also adds test for the extension.

commitextras: move fb extension to core which add extras to a commit This patch moves the Facebook extension to add extra fields to a commit to a in-core extension.

archive: add an experimental config to control the metadata file template Experimental because given the possible complexity, it may be worth figuring out how to load this from a file, similar to the style files for the log command, instead of trying to stuff it on the command line.

archive: use a templater to build the metadata file There are no visible changes here. I'm starting to wonder if adding the '+' to the 'node' line instead of a separate key line in 3047167733dc was the right thing to do. The '{node}' keyword never includes '+' elsewhere, and the way setup.py works, it would truncate it anyway. Additionally, the file is missing '{p2node}' when 'wdir()' merges are archived. I thought about adding an 'identify' line that would correspond to `hg id -n`. But the other nodes are the full 40 characters, and the output most useful for versioning is the short form. All of this cries out for customization via templating. (Although maybe having the short identify line by default is still a good idea.)

commandserver: do not handle EINTR for selector.select selectors2 library handles EINTR transparently so we don't need to handle select.error ourselves.

debugobsolete: also report the number of obsoleted changesets This seems useful to have the number of obsoleted changesets when calling debugobsolete.

transaction-summary: display the summary for all transactions Now that we records "all" changes happening in a transaction (in tr.changes) we will be able to provide better report on various changes (phases turned public, changeset obsoleted, branch merged or created, etc..) This is far too late in the cycle to play with this, but having this existing method called more widely will help extensions to play around with various options during the 4.4 cycle. Instead of calling registersummarycallback only for transactions we want, we always call it and use the transaction name to decide when to report (eg: we do not want `hg amend` to report new obsoleted changesets). Filtering on transaction name does not seems great, but seems good enough for the moment. We can change the API during the next cycle. The previous manual call during unbundling of the bundle2 "obsmarkers" part is no longer necessary and has been dropped.

test-strip: fix call to super class in crashstriprepo The 'self' argument is already taken care of by the 'super(...)' call. Instead we were passing the repository instance as the 'desc' value. This got discovered by breaking the next changeset.

share: share 'cachevfs' with the source clone (issue5108) Share extension now also share caches reads and writes. Not sharing caches results in costly caches recomputations which can takes up to minutes when using shares on large repositories. There are a couple of file in the '.hg/cache/' that depends of the current visibility. Visibility can be affected by the working copy location, something which is specific to each share. We ignores them for this series because they: * are the minority, * already have a good fallback to other precomputed caches, * are only affected when people use the experimental evolution feature.

cachevfs: add a devel warning for cache access though 'vfs' This will help third party extensions to migrate to the new 'cachevfs'.

cachevfs: migration the tags fnode cache to 'cachevfs' This will help sharing the cache between shares.

cachevfs: migrate tagscache to 'cachevfs' This will help sharing the cache between shares.

cachevfs: migration the revbranchcache to 'cachevfs' This will help sharing the cache between shares.

cachevfs: use the new vfs in when computing branchmap cache This will help sharing the cache between shares.

cachevfs: add a vfs dedicated to cache Most of the cache content lives in '.hg/cache/'. Moreover they are computed exclusively from data in the '.hg/store' directory. This creates issues with the share extension as the '.hg/store' directory is shared but the '.hg/cache' is not. On large repositories, this makes this prevent some usage of the share extension inefficient as some caches can take minutes to be recomputed. To improve the situation, we introduce a new 'cachevfs' that will be dedicated to cache reading and writing. In the next patches of this series, we'll migrate the 4 existing caches to it and update the share extension.

vfsward: register 'write with no lock' warnings as 'check-locks' config Update 'write with no lock' warnings in order to be better controlled by the config. We reuse the option used for lock order for these other lock related message. The message can now be disabled using 'devel.check-locks = no' (in addition to the usual 'devel.all-warnings = no').

demandimport: prefer loaded module over package attribute (issue5617) In general, the attribute of the same name is overwritten by executing an import statement. import a.b print(a.b.c) # 'c' of a/b/__init__.py from a.b.c import d print(a.b.c) # a/b/c.py However, this appears not true for the scenario described in the test case, and surprisingly, "from a.b.c import d" works even if "a.b.c" is not a module. This patch works around the problem by taking the right module from sys.modules if available.

demandimport: drop Py3 workarounds from Py2 implementation

demandimport: drop hack for old Pythons which had no level argument _import() was added by 868282fa29d8, but it is identical to _origimport() now.

tests: drop "absimport" conditionals which should be always true

color: drop the now useless color extension all the extension features are provided by core since 4.2.

extensions: expand the builtins extensions declaration This will future updates of the set cleaner and more readable.

debugextensions: drop the color from the test We are about to remove the extension so we remove trace of it in this specific test to prevent it to be confused.

configitems: register the 'bugzilla.notify' config

configitems: register the 'ui.mergemarkertemplate' config

configitems: register the 'ui.color' config

color: drop the _enabledbydefault module variable Since color is on by default, cleanup the now useless variable in both core and color extension.

configitems: register the 'ui.forcecwd' config

configitems: register the 'ui.fallbackencoding' config

bookmark: deprecate direct del of a bookmark value We want all bookmark deletion to go through 'applychanges', so lets deprecate legacy ways of doing bookmark deletion.

bookmark: deprecate direct set of a bookmark value We want all bookmark update to go through 'applychanges', so lets deprecate legacy ways of doing bookmark update.

bookmark: track bookmark changes at the transaction level The transaction has now a 'bookmarks' dictionary in tr.changes. The structure of the dictionary is {BOOKMARK_NAME: (OLD_NODE, NEW_NODE)}. If a bookmark is deleted NEW_NODE will be None. If a bookmark is created OLD_NODE will be None. If the bookmark is updated multiple time, the initial value is preserved.

bookmark: deprecate 'recordchange' in favor of 'applychanges' Now that we have migrated all in-core caller of 'recordchange' to 'applychanges', deprecate 'recordchange' so external callers will move to the new unified method.

bookmark: drop deletedivergent It has no caller anymore.

bookmark: use 'divergent2delete' in checkconflict checkconflict used to also do some bookmark deletion in case of divergence. It is a bit suspicious given the function name, but it's not the goal of this series. In order to unify bookmarks changing, checkconflict now return the list of divergent bookmarks to clean up and the callers must clean them by calling applyphases.

bookmark: use 'divergent2delete' when updating a bookmark

bookmark: use 'divergent2delete' in 'scmutil.cleanupnode'

bookmark: split out target computation from 'deletedivergent' We want to use applychanges in order to unify bookmark movement. We need a way to compute divergence deletion without actually removing them. We split the function in two in this patch while we migrate the existing users of this code on next patches.

bookmark: remove a useless 'recordchange' in the amend code We do not touch the bookmarks store in this code, just the active bookmark, not covered by the transaction. So it seems we can safely drop this call and the tests agree with us.

test-phase-exchange: stabilize for Windows

debugignore: eliminate inconsistencies with `hg status` (issue5222) Using a matcher for this command allows processing the named file(s) as relative to cwd. It also leverages the icasefs normalization logic the same way the status command does. (However, a false indicator is given for a nonexistent file in some cases, e.g. passing 'foo.REJ' when that file doesn't exist, and the rule is '*.rej'. Maybe the regex itself needs to be case insensitive on these platforms, at least for the debug command.) Finally, the file printed is relative to cwd and uses platform specific slashes, so a few (glob)s were needed in seemingly unrelated tests.

commandserver: close selector explicitly The selector does not have a __del__ method and needs a manual close. We can also use "with selector" but that makes the code too indented. Therefore append a "selector.close()" after the end of the main loop for now.

scmutil: remove duplicated import of i18n._()

obsstore: let read marker API take a range of offsets This allows us to read a customized range of markers, instead of loading all of them. The condition of stop is made consistent across C and Python implementation so we will still read marker when offset=a, stop=a+1.

commandserver: use selectors2 Previously, commandserver was using select.select. That could have issue if _sock.fileno() >= FD_SETSIZE (usually 1024), which raises: ValueError: filedescriptor out of range in select() We got that in production today, although it's the code opening that many files to blame, it seems better for commandserver to work in this case. There are multiple way to "solve" it, like preserving a fd with a small number and swap it with sock using dup2(). But upgrading to a modern selector supported by the system seems to be the most correct way.

selector2: vendor selector2 library This library was a backport of the Python 3 "selectors" library. It is useful to provide a better selector interface for Python2, to address some issues of the plain old select.select, mentioned in the next patch. The code [1] was ported using the MIT license, with some minor modifications to make our test happy: 1. "# no-check-code" was added since it's foreign code. 2. "from __future__ import absolute_import" was added. 3. "from collections import namedtuple, Mapping" changed to avoid direct symbol import. [1]: https://github.com/SethMichaelLarson/selectors2/blob/d27dbd2fdc48331fb76ed431f44b6e6956de7f82/selectors2.py # no-check-commit

context: name files relative to cwd in warning messages I was several directories deep in the kernel tree, ran `hg add`, and got the warning about the size of one of the files. I noticed that it suggested undoing the add with a specific revert command. The problem is, it would have failed since the path printed was relative to the repo root instead of cwd. While here, I just fixed the other messages too. As an added benefit, these messages now look the same as the verbose/inexact messages for the corresponding command. I don't think most of these messages are reachable (typically the corresponding cmdutil function does the check). I wasn't able to figure out why the keyword tests were failing when using pathto()- I couldn't cause an absolute path to be used by manipulating the --cwd flag on a regular add. (I did notice that keyword is adding the file without holding wlock.)

run-tests: disable color on Windows More Windows sadness. Maybe someone can figure out how to make win32 color work, but I think we avoid importing stuff from the mercurial package in this module. On the plus side, this conditionalizes away a test failure.

codemod: register core configitems using a script This is done by a script [2] using RedBaron [1], a tool designed for doing code refactoring. All "default" values are decided by the script and are strongly consistent with the existing code. There are 2 changes done manually to fix tests: [warn] mercurial/exchange.py: experimental.bundle2-output-capture: default needs manual removal [warn] mercurial/localrepo.py: experimental.hook-track-tags: default needs manual removal Since RedBaron is not confident about how to indent things [2]. [1]: https://github.com/PyCQA/redbaron [2]: https://github.com/PyCQA/redbaron/issues/100 [3]: #!/usr/bin/env python # codemod_configitems.py - codemod tool to fill configitems # # Copyright 2017 Facebook, Inc. # # This software may be used and distributed according to the terms of the # GNU General Public License version 2 or any later version. from __future__ import absolute_import, print_function import os import sys import redbaron def readpath(path): with open(path) as f: return f.read() def writepath(path, content): with open(path, 'w') as f: f.write(content) _configmethods = {'config', 'configbool', 'configint', 'configbytes', 'configlist', 'configdate'} def extractstring(rnode): """get the string from a RedBaron string or call_argument node""" while rnode.type != 'string': rnode = rnode.value return rnode.value[1:-1] # unquote, "'str'" -> "str" def uiconfigitems(red): """match *.ui.config* pattern, yield (node, method, args, section, name)""" for node in red.find_all('atomtrailers'): entry = None try: obj = node[-3].value method = node[-2].value args = node[-1] section = args[0].value name = args[1].value if (obj in ('ui', 'self') and method in _configmethods and section.type == 'string' and name.type == 'string'): entry = (node, method, args, extractstring(section), extractstring(name)) except Exception: pass else: if entry: yield entry def coreconfigitems(red): """match coreconfigitem(...) pattern, yield (node, args, section, name)""" for node in red.find_all('atomtrailers'): entry = None try: args = node[1] section = args[0].value name = args[1].value if (node[0].value == 'coreconfigitem' and section.type == 'string' and name.type == 'string'): entry = (node, args, extractstring(section), extractstring(name)) except Exception: pass else: if entry: yield entry def registercoreconfig(cfgred, section, name, defaultrepr): """insert coreconfigitem to cfgred AST section and name are plain string, defaultrepr is a string """ # find a place to insert the "coreconfigitem" item entries = list(coreconfigitems(cfgred)) for node, args, nodesection, nodename in reversed(entries): if (nodesection, nodename) < (section, name): # insert after this entry node.insert_after( 'coreconfigitem(%r, %r,\n' ' default=%s,\n' ')' % (section, name, defaultrepr)) return def main(argv): if not argv: print('Usage: codemod_configitems.py FILES\n' 'For example, FILES could be "{hgext,mercurial}/*/**.py"') dirname = os.path.dirname reporoot = dirname(dirname(dirname(os.path.abspath(__file__)))) # register configitems to this destination cfgpath = os.path.join(reporoot, 'mercurial', 'configitems.py') cfgred = redbaron.RedBaron(readpath(cfgpath)) # state about what to do registered = set((s, n) for n, a, s, n in coreconfigitems(cfgred)) toregister = {} # {(section, name): defaultrepr} coreconfigs = set() # {(section, name)}, whether it's used in core # first loop: scan all files before taking any action for i, path in enumerate(argv): print('(%d/%d) scanning %s' % (i + 1, len(argv), path)) iscore = ('mercurial' in path) and ('hgext' not in path) red = redbaron.RedBaron(readpath(path)) # find all repo.ui.config* and ui.config* calls, and collect their # section, name and default value information. for node, method, args, section, name in uiconfigitems(red): if section == 'web': # [web] section has some weirdness, ignore them for now continue defaultrepr = None key = (section, name) if len(args) == 2: if key in registered: continue if method == 'configlist': defaultrepr = 'list' elif method == 'configbool': defaultrepr = 'False' else: defaultrepr = 'None' elif len(args) >= 3 and (args[2].target is None or args[2].target.value == 'default'): # try to understand the "default" value dnode = args[2].value if dnode.type == 'name': if dnode.value in {'None', 'True', 'False'}: defaultrepr = dnode.value elif dnode.type == 'string': defaultrepr = repr(dnode.value[1:-1]) elif dnode.type in ('int', 'float'): defaultrepr = dnode.value # inconsistent default if key in toregister and toregister[key] != defaultrepr: defaultrepr = None # interesting to rewrite if key not in registered: if defaultrepr is None: print('[note] %s: %s.%s: unsupported default' % (path, section, name)) registered.add(key) # skip checking it again else: toregister[key] = defaultrepr if iscore: coreconfigs.add(key) # second loop: rewrite files given "toregister" result for path in argv: # reconstruct redbaron - trade CPU for memory red = redbaron.RedBaron(readpath(path)) changed = False for node, method, args, section, name in uiconfigitems(red): key = (section, name) defaultrepr = toregister.get(key) if defaultrepr is None or key not in coreconfigs: continue if len(args) >= 3 and (args[2].target is None or args[2].target.value == 'default'): try: del args[2] changed = True except Exception: # redbaron fails to do the rewrite due to indentation # see https://github.com/PyCQA/redbaron/issues/100 print('[warn] %s: %s.%s: default needs manual removal' % (path, section, name)) if key not in registered: print('registering %s.%s' % (section, name)) registercoreconfig(cfgred, section, name, defaultrepr) registered.add(key) if changed: print('updating %s' % path) writepath(path, red.dumps()) if toregister: print('updating configitems.py') writepath(cfgpath, cfgred.dumps()) if __name__ == "__main__": sys.exit(main(sys.argv[1:]))

phabricator: allow specifying reviewers on phabsend Sometimes people want to specify reviewer explicitly for a stack. The webpage only allows changing reviewer for one revision at a time. This patch adds a `--reviewer` flag to make it easier to specify reviewers. Test Plan: On a test Phabricator instance, enable `differential.allow-self-accept`, assign myself as a reviewer and make sure it works. Also try an invalid username and make sure it raises. Differential Revision: https://phab.mercurial-scm.org/D38

match: remove unused negatematcher This was only used by the sparse extension's dirstate._ignore override, which no longer exists. Differential Revision: https://phab.mercurial-scm.org/D60

sparse: override dirstate.walk() instead of dirstate._ignore Instead of treating files that are outside the sparse config as ignored, this makes it so we list only those that are within the sparse config by passing the sparse matcher to dirstate.walk(). Once we add support for narrow (sparseness applied to history, not just working copy), we will need to do a similar restriction of the walk over manifests, so this will be more consistent then. It also simplifies the code a bit. Note that a side-effect of this change is that files outside the sparse config used to be listed as ignored, but they will now not be listed at all. This can be seen in the test case where "hg purge" no longer has any effect because it doesn't see that the files outside the space config exist. To fix that, I think we should add an option to dirstate.walk() to walk outside the sparse config. We might expose that to the user as --no-sparse flag to e.g. "hg status" and "hg purge", but that's work for another day. Differential Revision: https://phab.mercurial-scm.org/D59

patch: use devel.all-warnings to replace devel.all It appears to be a misspell in patch.py.

sslutil: inform the user about how to fix an incomplete certificate chain This is a Windows only thing. Unfortunately, the socket is closed at this point (so the certificate is unavailable to check the chain). That means it's printed out when verification fails as a guess, on the assumption that 1) most of the time verification won't fail, and 2) sites using expired or certs that are too new will be rare. Maybe this is an argument for adding more functionality to debugssl, to test for problems and print certificate info. Or maybe it's an argument for bundling certificates with the Windows builds. That idea was set aside when the enhanced SSL code went in last summer, and it looks like there were issues with using certifi on Windows anyway[1]. This was tested by deleting the certificate out of certmgr.msc > "Third-Party Root Certification Authorities" > "Certificates", seeing `hg pull` fail (with the new message), trying this command, and then successfully performing the pull command. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/089573.html

debug: add a method to check the state of, and built an SSL cert chain This is only useful on Windows, and avoids the need to use Internet Explorer to build the certificate chain. I can see this being extended in the future to print information about the certificate(s) to help debug issues on any platform. Maybe even perform some of the python checks listed on the secure connections wiki page. But for now, all I need is 1) a command that can be invoked in a setup script to ensure the certificate is installed, and 2) a command that the user can run if/when a certificate changes in the future. It would have been nice to leverage the sslutil library to pick up host specific settings, but attempting to use sslutil.wrapsocket() failed the 'not sslsocket.cipher()' check in it and aborted. The output is a little more chatty than some commands, but I've seen the update take 10+ seconds, and this is only a debug command.