match: explicitly naming a subrepo implies always() for the submatcher The files command supports naming directories to limit the output to children of that directory, and it also supports -S to force recursion into a subrepo. But previously, using -S and naming a subrepo caused nothing to be output. The reason was narrowmatcher() strips the current subrepo path off of each file, which would leave an empty list if only the subrepo was named. When matching on workingctx, dirstate.matches() would see the submatcher is not always(), so it returned the list of files in dmap for each file in the matcher- namely, an empty list. If a directory in a subrepo was named, the output was as expected, so this was inconsistent. The 'not anypats()' check is enforced by an existing test around line 140: $ hg remove -I 're:.*.txt' sub1 Without the check, this removed all of the files in the subrepo.

context: don't complain about a matcher's subrepo paths in changectx.walk() Previously, the first added test printed the following: $ hg files -S -r '.^' sub1/sub2/folder sub1/sub2/folder: no such file in rev 9bb10eebee29 sub1/sub2/folder: no such file in rev 9bb10eebee29 sub1/sub2/folder/test.txt One warning occured each time a subrepo was crossed into. The second test ensures that the matcher copy stays in place. Without the copy, the bad() function becomes an increasingly longer chain, and no message would be printed out for a file missing in the subrepo because the predicate would match in one of the replaced methods. Manifest doesn't know anything about subrepos, so it needs help ignoring subrepos when complaining about bad files.

ssh: capture output with bundle2 again (issue4642) I just discovered that we are not displaying ssh server output in real time anymore. So we can just fall back to the bundle2 output capture for now. This fix the race condition issue we where seeing in tests. Re-instating real time output for ssh would fix the issue too but lets get the test to pass first.

revset: optimize not public revset This patvh speeds up the computation of the not public() changeset and incidentally speed up the computation of divergents() changeset on our big repo by 100x from 50% to 0.5% of the time spent in smartlog with evolve. In this patch we optimize not public() to _notpublic() (new revset) and use the work on phaseset (from the previous commit) to be able to compute _notpublic() quickly. We use a non-lazy approach making the assumption the number of notpublic change will not be in the order of magnitude of the repo size. Adopting a lazy approach gives a speedup of 5x (vs 100x) only due to the overhead of the code for lazy generation.

phases: add set per phase in C phase computation To speed up the computation of draft(), secret(), divergent(), obsolete() and unstable() we need to have a fast way of getting the list of revisions that are in draft(), secret() or the union of both: not public(). This patch extends the work on phase computation in C and make the phase computation code also return a list of set for each non public phase. Using these sets we can quickly obtain all the revisions of a given phase. We do not return a set for the public phase as we expect it to be roughly the size of the repo. Also, it can be computed easily by substracting the entries in the non public phases from all the revs in the repo.

match: rename _fmap to _fileroots for clarity fmap isn't a very descriptive name for the set of the match's files.

match: remove unnecessary optimization where visitdir() returns 'all' Match's visitdir() was prematurely optimized to return 'all' in some cases, so that the caller would not have to call it for directories within the current directory. This change makes the visitdir system less flexible for future changes, such as making visitdir consider the match's include and exclude patterns. As a demonstration of this optimization not actually improving performance, I ran 'hg files -r . media' on the Mozilla repository, stored as treemanifest revlogs. With best of ten tries, the command took 1.07s both with and without the optimization, even though the optimization reduced the calls from visitdir() from 987 to 51.

dispatch: add support for python-flamegraph[0] profiling This gives us nicer svg flame graphs for output, which can make understanding some types of performance problems significantly easier. 0: https://github.com/evanhempel/python-flamegraph/

extensions: document that `testedwith = 'internal'` is special Extension authors (notably at companies using hg) have been cargo-culting the `testedwith = 'internal'` bit from hg's own extensions, which then defeats our "file bugs over here" logic in dispatch. Let's be more aggressive about trying to give extension authors a hint about what testedwith should say.

treemanifest: cache directory logs and manifests Since manifests instances are cached on the manifest log instance, we can cache directory manifests by caching the directory manifest logs. The directory manifest log cache is a plain dict, so it never expires; we assume that we can keep all the directories in memory. The cache is kept on the root manifestlog, so access to directory manifest logs now has to go through the root manifest log. The caching will soon not be only an optimization. When we start lazily loading directory manifests, we need to make sure we don't create multiple instances of the log objects. The caching takes care of that problem.

hook: drop dedicated catch for 'KeyboardInterrupt' This is no longer under 'Exception' in Python 2.6.

recover: catch any exception, not just Exception We want recover to be rock solid.

exchange: catch down to BaseException when handling bundle2 We can now catch more things.

bundle2: use BaseException in bundle2 We can ensure we fail over properly in more cases.

check-code: drop ban of BaseException Lets go back to the basic. It is available in Python 2.6.

wireproto: turn an 'except' into a 'finally' as suggest by the comment Look! More hidden footprints!

check-code: drop the yield inside try/finally ban This is now possible with Python 2.6.

run-tests: resurrect the wifexited polyfill (backout 6ab5a1c9ea3c) Python 2.7.3 on Windows doesn't have os.WIFEXITED, and the test output looked like this before I interrupted it. $ ./run-tests.py --local -j2 -t700 EEEEEEEEEEEEEEEEEEEEEE

tests: check import cycles in hgext/**.py, too It is important to realize existing cycles in hgext/**.py.

import-checker: don't treat modules as relative one if not found The previous patch ensures all module names are recorded in `imports` as absolute names, so we no longer need to treat modules as ones imported relatively from the target source if they appear to not be from the stdlib.

import-checker: make imported_modules yield absolute dotted_name_of_path This patch makes `imported_modules()` always yield absolute `dotted_name_of_path()`-ed name by strict detection with `fromlocal()`. This change improves circular detection in some points: - locally defined modules, of which name collides against one of standard library, can be examined correctly For example, circular import related to `commands` is overlooked before this patch. - names not useful for circular detection are ignored Names below are also yielded before this patch: - module names of standard library (= not locally defined one) - non-module names (e.g. `node.nullid` of `from node import nullid`) These redundant names decrease performance of circular detection. For example, with files at 1ef96a3b8b89, average loops per file in `checkmod()` is reduced from 165 to 109. - `__init__` can be handled correctly in `checkmod()` For example, current implementation has problems below: - `from xxx import yyy` doesn't recognize `xxx.__init__` as imported - `xxx.__init__` imported via `import xxx` is treated as `xxx`, and circular detection is aborted, because `key` of such module name is not `xxx` but `xxx.__init__` - it is easy to enhance for `from . import xxx` style or so (in the future) Module name detection in `imported_modules()` can use information in `ast.ImportFrom` fully. It is assumed that all locally defined modules are correctly specified to `import-checker.py` at once. Strictly speaking, when `from foo.bar.baz import module1` imports `foo.bar.baz.module1` module, current `imported_modules()` yields only `foo.bar.baz.__init__`, even though also `foo.__init__` and `foo.bar.__init__` should be yielded to detect circular import exactly. But this limitation is reasonable one for improvement in this patch, because current `__init__` files in Mercurial seems to be implemented carefully.

import-checker: add utility to examine what module is imported easily `fromlocalfunc()` uses: - `modulename` (of the target source) to compose absolute module name imported relatively from it It is assumed that `modulename` is an `dotted_name_of_path()`-ed source file, which may have `.__init__` at the end of it. This assumption makes composing `prefix` of relative name easy. - `localmods` to examine whether there is a locally defined (= Mercurial specific) module matching against the specified name It is assumed that module names not existing in `localmods` are ones of Python standard library.

subrepo: further replacement of try/except with 'next' Burn StopIteration, Burn!

parsers: use 'next' instead of try/except This get rid of another StopIteration abomination. The change in self.current value is supposed to not matter as nobody should be calling '_advance' after that (as per Matt wisdom).

dagparser: use 'next' instead of try/except for default value "Ich liebe es wenn ein Plan funktioniert."

getlogrevs: rewrite a loop to get read of try/except Get rid of the 'except StopIteration' abomination.

_makelogrevset: replace try/except with 'next' usage More readable without the 'except StopIteration' abomination.

ignore: move readpatternfile to match.py In preparation for adding 'include:' rule support to match.py, let's move the pattern file reader function to match.py

ignore: rename readignorefile to readpatternfile A future commit will move the readignorefile logic into match.py so it can be used from general match rules. Let's rename the function to represent this new behavior.

ignore: combine readignorefile and _ignorefile _ignorefile did nothing except open the file. Let's combine it with readignorefile for simplicity. This will make it easier to rename and move to match.py in upcoming patches.

ignore: move bad file handling out of readignorefile In preparation for moving readignorefile to match.py to make it more generally usable, let's move the bad ignore file handling up to the ignore specific logic.

ignore: remove .hgignore from ignore list if nonexistent Previously we would always pass the root .hgignore path to the ignore parser. The parser then had to be aware that the first path was special, and not warn if it didn't exist. In preparation for making the ignore file parser more generically usable, let's make the parse logic not aware of this special case, and instead just not pass the root .hgignore in if it doesn't exist.

run-tests: replace open-coded .decode()s on paths with a helper (issue4667) This also cleans up the mkdtemp code mentioned in the previous patch. At this point, the remaining callsites of .{en,de)code() are in the following categories: Handling escaped lines in .t files ----------------------------------- It seems eminently reasonable to me for us to declare that .t files are valid utf-8, and that any escape sequences we see in .t files should be valid unicode_escape sequences. Making error text safe for cdata blocks for xml error reports ------------------------------------------------------------- This is a point where we're already basically screwed, and we're simply trying to do something "good enough" that the xml output will be vaguely useful to the user. Punting here seems fine, and we should probably stick to the same encoding here that we used in the previous section.

run-tests: move unicode-to-bytes operations on paths to a helper (issue4667) This doesn't fix the probably-wrong utf-8 encoding choice, it just starts the process of encapsulating all the path handling in run-tests in a single place. One known-path use of .encode() remains: it's related to use of mkdtemp, and it will be fixed in a followup patch once we have a companion _strpath() helper function to go from bytes to a str, as we need to file a bug about mkdtemp upstream.

run-tests: insist that if people use Python 3, they use 3.5.x We depend on both stdlib functionality (difflib.diff_bytes) and language behavior (bytes formatting) introduced in 3.5, so let's try and prevent some useless bug reports before they happen.

run-tests: move all open-coded sys.version_info checks to PYTHON3 (issue4668) This consolidates all version checking in a single place and helps the code read more obviously.

run-tests: prefer PYTHON3 constant to many version_info checks (issue4668) We only support Python 2.6, 2.7, and 3.5 here, so we can simplify the code and improve the warning.

run-tests: introduce PYTHON3 boolean constant (issue4668) This will avoid open-coding lots of version_info checks later in the file.

run-tests: drop subprocess _cleanup monkeypatch This was working around a defect in subprocess in Python 2.5, which we no longer need to worry about.

run-tests: drop wifexited polyfill os.WIFEXITED exists as of python 2.6 for me, so we no longer need this polyfill.

test-import: update for no-execbit platforms with 1ef96a3b8b89

test-treemanifest: add globs for Windows

util: drop any() and all() polyfills

cleanup: use __builtins__.all instead of util.all

check-code: un-ban __builtins__.all now that we're on 2.6

cleanup: use __builtins__.any instead of util.any any() is available in all Python versions we support now.

check-code: un-ban any() now that we're on 2.6

walkchangerevs: replace try/except with 'next' Again, this make the code clearer.

generatorset: use 'next()' to simplify the code The 'next()' built-in accept a default value. This remove the needs to check if self non-empty before returning a value.

revset: use 'next()' to detect end of iteration in 'last' The 'next()' built-in can return a default value, allow to get rid of the confusing try/except code flow.

revset: use 'next()' to detect end of iteration in 'limit' The 'next()' built-in can return a default value, allow to get rid of the confusing try/except code flow.

_revancestors: use 'next' to remove the verbose try except clauses The 'next()' built-in can return a default value, making the final iteration case simpler and clearer.

check-code: lift the ban on 'next()' 'next' is supported by Python2.6 the new lowest version supported by Mercurial.

check-code: drop the 'isdisjoint' ban 'isdisjoint' is available in Python 2.6. The new lowest supported version.

check-code: fix the error message about 'class foo():' Using 'classs foo():' result in old style object instead of new style object. We do not want old style object so this check is unrelated to 2.4.

check-code: remove the check for os.path.relpath This is available in Python2.6 the new default supported release.

patch: remove duplicated import of cStringIO It is already imported a couple of line above.

check-commit: print limit when user has a too-long summary

paper: show branch/tags/bookmarks when blaming (issue3559)

paper: show branch/tags/bookmarks when comparing (issue3559)

paper: show branch/tags/bookmarks when diffing (issue3559)

paper: show branch/tags/bookmarks when viewing (issue3559)

paper: show branch/tags/bookmarks when browsing (issue3559) Browse (or manifest) action allows browsing the directory structure at some specified revision. In gitweb and monoblue styles, the revision header already has branch/tag/bookmark information for the revision, but in paper style this header was only showing tags. This patch adds branches and bookmarks. Branch name needs to be obtained in this special way to be consistent with regular changeset page, where in paper style default branch is never shown.

revset: extract addset._iterordered to free function It never uses self, so let's make it less dependent on variables.

revset: use fastasc/fastdesc switch consistently in addset.__iter__

revset: drop redundant filteredset from right-hand side set of "or" operation Since 5548f558db3d, it should no longer be necessary because the addset can remove duplicates correctly.

getbundle: sort bundlecaps before exchanging then over the wire The 'bundlecaps' argument is built as a set, we need to stabilise the order before exchanging them. Otherwise, in the test, http logs are unstable when the 'bundlecaps' contains something (eg: using bundle2).

hg-ssh: reject push earlier (on pretxnopen) We now have a lock triggered for any transaction. We use it to ensure no-read are made in read-only mode. We need more that just "no changegroup is added", since bundle2 allows for more than just changegroup to be exchanged. We still protect pushkey as it may write data without opening a transaction.

test: test both bundle1 and bundle2 paths in 'test-push-hook-lock.t' The locking scheme is a bit different in each cases. So we explicitly test each case.

progress: get the extremely verbose output out of default debug When the progress extension is not enabled, each call to 'ui.progress' used to issue a debug message. This results is a very verbose output and often redundant in tests. Dropping it makes tests less volatile to factor they do not meant to test. We had to alter the sed trick in 'test-rename-merge2.t'. Sed is used to drop all output from a certain point and hidding the progress output remove its anchor. So we anchor on something else.

hgweb: show changeset branches/tags/bookmarks in file log (style=monoblue) As for the gitweb style, this line for filelogentry template is copied from shortlogentry. No change to python code is needed. Tests are unaffected.

hgweb: show changeset branches/tags/bookmarks in file log (style=gitweb) This line for filelogentry template is copied from shortlogentry. The information is already displayed in paper style, so no change to python code is needed. Tests are unaffected.

match: resolve filesets in subrepos for commands given the '-S' argument This will work for any command that creates its matcher via scmutil.match(), but only the files command is tested here (both workingctx and basectx based tests). The previous behavior was to completely ignore the files in the subrepo, even though -S was given. My first attempt was to teach context.walk() to optionally recurse, but once that was in place and the complete file list was built up, the predicate test would fail with 'path in nested repo' when a file in a subrepo was accessed through the parent context. There are two slightly surprising behaviors with this functionality. First, any path provided inside the fileset isn't narrowed when it is passed to the subrepo. I dont see any clean way to do that in the matcher. Fortunately, the 'subrepo()' fileset is the only one to take a path. The second surprise is that status predicates are resolved against the subrepo, not the parent like 'hg status -S' is. I don't see any way to fix that either, given the path auditor error mentioned above.

subrepo: introduce getfileset() This will be used in the next patch to help matchers resolve filesets in subrepos. The default implementation returns an empty set (for git and svn).

phase: default to current revision if no rev is provided (issue4666)

treemanifest: add --dir option to debug{revlog,data,index} It should be possible to debug the submanifest revlogs without having to know where they are stored (in .hg/store/meta/), so let's add a --dir option for this purpose.

obsolete: sort obsmarkers during exchange Because bundle2 allows a more precise exchange of obsmarkers during pull, it sends them in a different order (previously unstable because of sets.) As a result, they are added to the repository in a different order. To stabilize the order and ensure tests are unchanged when moving from bundle1 to bundle2 we sort markers when exchanging them. In the long run, the obsstore will probably not use a linear storage.

test-discovery-legacy: disable bundle2 for legacy test We need to disable the "bundle2" capability to ensure we stay on legacy capability.

test-discovery: disable bundle2 for legacy test (issue4610) We need to disable the "bundle2" capability to ensure we stay on legacy capability.

revset: fix iteration over ordered addset composed of non-ordered operands Before this change, doing ordered iteration over an 'addset' object composed of operands without fastasc or fastdesc method could result in duplicated entries. This was the result of applying '_iterordered' on an unordered set. We fix it by ensuring we iterate over the set in a sorted order. Using the fast iterator when it exists on any operand. We kill the '_iterator' method in the process because it did not make a lot of sense independently. Thanks goes to Yuya Nishihara for reporting the issue and analysing the cause.

match: add match.ispartial() match.ispartial() will return the opposite of match.always() in core, but this function will be extensible by extensions to produce another result even if match.always() will be untouched. This will be useful for narrowhg, where ispartial() will return False even if the match won't always match. This would happen in the case where the only time the match function is False is when the path is outside of the narrow spec.

util: drop alias for collections.deque Now that util.deque is just an alias for collections.deque, let's just remove it.

util: kill Python 2.4 deque.remove hack

summary: add a phase line (draft, secret) to the output The number of draft and secret changesets are currently not summarized. This is an important information because the number of drafts give some rough idea of the number of outgoing changesets in typical workflows, without needing to probe a remote repository. And a non-zero number of secrets means that those changeset will not be pushed. If the repository is "dirty" - some draft or secret changesets exists - then summary will display a line like: phases: X draft, Y secret (public) The phase in parenthesis corresponds to the highest phase of the parents of the working directory, i.e. the current phase. By default, the line is not printed if the repository is "clean" - all changesets are public - but if verbose is activated, it will display: phases: (public) On the other hand, nothing will be printed if quiet is in action. A few tests have been added in test-phases.t to cover the -v and -q cases.

hgweb: also display changeset bookmarks in file log (style=paper) There already are branches and tags in file log, now let's add what's been missing: bookmarks. Also, since coal borrows this template from paper, this change is effective for coal as well.

tests: test branch/tag/bookmark display in test-hgweb-filelog.t Displaying branches, tags and bookmarks is an obviously important feature of hgweb and should be tested a bit more than not at all, so let's add a branch, a tag and a bookmark to the test. With this change it's evident that the default style (paper) doesn't show bookmarks in filelog. Future patch will fix this.

hgweb: split some long lines in paper, gitweb and monoblue styles This will ease future patches for the templates. As a result of this patch, paper style has one visual change in log/shortlog/file log view: the spacing between commit message and the first tag (or branch name, or bookmark) is now roughly who spaces wide instead of one space wide. This spacing is consistent with the one between branch names/tags/bookmarks themselves, so it looks better. In gitweb style, the change from non-breakable space to regular space is consistent with other elements. In monoblue the change is not noticeable.

run-tests: add a --profile-runner option While in the throes of a recent run-tests adventure, I found it useful to have profiler output for the testrunner itself. Adding it was simple enough and seems worth keeping around.

test-http: drop compatibility check on Python >= 2.4.3 This is obsolete now that Python < 2.6 is no more supported. This is thus a backout of c499fff76440.

revset: remove unused 'only' from methods table The infix 'only' operator is mapped to 'only()' function by optimize(), so it won't be looked up as a method. The test shows it.

shelve: allow --patch and --stat without --list for a single shelf It's annoying having to specify --list and --patch/--stat when all you really want to do is to dump a patch. This creates an explicit --patch/--stat command that is executed if --list is not specified. It ensures that 1) there is only one shelf name specified and 2) that the shelf exists. Then it redirects to the original listcmd code.

shelve: refactor allowables to specify sets of valid operations This will allow us to have --patch and --stat as standalone operations that don't require --list, as well as pairing them with --list.

merge with stable

rebase: check that the bookmark is still valid when restoring (issue4669) After a rebase --abort, we attempt to restore the previously active bookmark. We need to ensure that the bookmark still exists.

bookmarks: rename current to active in variables and comments Today, the terms 'active' and 'current' are interchangeably used throughout the codebase in reference to the active bookmark (the bookmark that will be updated with the next commit). This leads to confusion among developers and users. This patch is part of a series to standardize the usage to 'active' throughout the mercurial codebase and user interface.

rebase: remove extraneous blank at the end of file

run-tests: include 'start' and 'end' in --time output This is useful information to understand what is taking time in tests. Both are included because I can see myself sorting this output using shell script. Having both data makes it much easier than extracting 'start'+'real'.

run-tests: track start and end time of tests We currently have information about how long each test took, but we have no data about their actual scheduling. So we now track when a test started and stopped (in the referential of the whole tests run) to expose this information. The data is currently in the json only because the json output is meant to be extensible. Later changeset will includes this data in the text output and we will be happy to bikeshed its formating there. Yes, "end" is actually just "start" + "time", but computing it an including it in the output is simple, cheap and convenient.

templater: look for mapfiles in template paths This will allow %include statements to search the default template paths in addition to the directory where the %including file is.

config: give it an includepaths option for looking for config files It is desirable to "derive" templates from the provided templates. A simple way to do this is e.g. %include map-cmdline.default in your own mapfile. Then you only have to redefine a few templates instead of copying over the whole thing. This %include mechanism already works for the built-in templates because by default it *only* looks for files that are in the same directory as the including mapfile. With this changeset, config grows an option to add more include paths for config files.

revset: map postfix '%' to only() to optimize operand recursively (issue4670) Instead of keeping 'onlypost' as a method, this patch rewrites it to 'only' function. This way, 'x%' always has the same weight as 'only(x)'.

dirs.c: pass C string, not Python string, to _finddir() The callers already have the C string, and although the PyString_AS_STRING() macro is probably free, this simplifies the code.

dirs.c: extract 'cpath' variable in _delpath() to match _addpath() The PyString_AS_STRING() macro is probably free, but this makes _delpath() more similar to _addpath() and simplifies the next patch.

treemanifest: store submanifest revlog per directory With this change, when tree manifests are enabled (in .hg/requires), commits will be written with one manifest revlog per directory. The manifest revlogs are stored in .hg/store/meta/$dir/00manifest.[id]. Flat manifests can still be read and interacted with as usual (they are also read into treemanifest instances). The functionality for writing treemanifest as a flat manifest to disk is still left in the code; tests still pass with '_treeinmem=True' hardcoded. Exchange is not yet implemented.

hghave: use try/except/finally

setup: use try/except/finally This will raise a syntax error for people who attempt to use Py2.4, but that's already going to fail and we have no way to keep other 2.6isms from creeping in since we've removed the check-code rules and the buildbot.

util: use try/except/finally

tags: use try/except/finally

repoview: use try/except/finally

httppeer: use try/except/finally

hooks: use try/except/finally

hgweb: use try/except/finally

commands: use try/except/finally

bookmarks: use try/except/finally

shelve: use try/except/finally

largefiles: use try/except/finally

factotum: use try/except/finally

check-code: drop try/except/finally check

util.h: kill no longer needed definitions for Python < 2.6 see e1fb276d4619

merge with stable