treemanifest: remove treemanifest._intersectfiles() In preparation for the optimization in the following commit, this commit removes treemanifest.matches()'s call to _intersectfiles(), and removes _intersectfiles() itself since it's unused at this point.

manifest: add some tests for manifest.matches() There were no tests for the various code paths in manifestdict.matches(), so I added some. This also adds a more complex testing manifest so that any bugs relating to traversal of directories are more likely to be caught.

forget: cleanup the output for an inexact case match on icasefs Previously, specifying a file name but not matching the dirstate case yielded the following, even though the file was actually removed: $ hg forget capsdir1/capsdir/abc.txt not removing capsdir\a.txt: file is already untracked removing CapsDir\A.txt [1] This change doesn't appear to cause any extra filesystem accesses, even if a nonexistant file is specified. If a directory is specified without a case match, it is (and was previously) still silently ignored.

json: implement {branches} template

json: implement {bookmarks} template

json: implement {tags} template Tags is pretty easy to implement. Let's start there. The output is slightly different from `hg tags -Tjson`. For reference, the CLI has the following output: [ { "node": "e2049974f9a23176c2addb61d8f5b86e0d620490", "rev": 29880, "tag": "tip", "type": "" }, ... ] Our output has the format: { "node": "0aeb19ea57a6d223bacddda3871cb78f24b06510", "tags": [ { "node": "e2049974f9a23176c2addb61d8f5b86e0d620490", "tag": "tag1", "date": [1427775457.0, 25200] }, ... ] } "rev" is omitted because it isn't a reliable identifier. We shouldn't be exposing them in web APIs and giving the impression it remotely resembles a stable identifier. Perhaps we could one day hide this behind a config option (it might be useful to expose when running servers locally). The "type" of the tag isn't defined because this information isn't yet exposed to the hgweb templater (it could be in a follow-up) and because it is questionable whether different types should be exposed at all. (Should the web interface really be exposing "local" tags?) We use an object for the outer type instead of Array for a few reasons. First, it is extensible. If we ever need to throw more global properties into the output, we can do that without breaking backwards compatibility (property additions should be backwards compatible). Second, uniformity in web APIs is nice. Having everything return objects seems much saner than a mix of array and object. Third, there are security issues with arrays in older browsers. The JSON web services world almost never uses arrays as the main type for this reason. Another possibly controversial part about this patch is how dates are defined. While JSON has a Date type, it is based on the JavaScript Date type, which is widely considered a pile of garbage. It is a non-starter for this reason. Many of Mercurial's built-in date filters drop seconds resolution. So that's a non-starter as well, since we want the API to be lossless where possible. rfc3339date, rfc822date, isodatesec, and date are all lossless. However, they each require the client to perform string parsing on top of JSON decoding. While date parsing libraries are pretty ubiquitous, some languages don't have them out of the box. However, pretty much every programming language can deal with UNIX timestamps (which are just integers or floats). So, we choose to use Mercurial's internal date representation, which in JSON is modeled as float seconds since UNIX epoch and an integer timezone offset from UTC (keep in mind JavaScript/JSON models all "Numbers" as double prevision floating point numbers, so there isn't a difference between ints and floats in JSON).

templates: add a stub template for json Many have long wanted hgweb to emit a common machine readable output. We start the process by defining a stub json template. Right now, each endpoint returns a stub "not yet implemented" string. Individual templates will be implemented in subsequent patches. Basic tests for templates have been included. Coverage isn't perfect, but it is better than nothing.

get-with-headers: support parsing and pretty printing JSON Upcoming patches will add support for JSON output from hgweb. Because JSON output from the templater is hard to read and because it is easy to introduce malformed JSON, we introduce a JSON processing mode to get-with-headers.py that will parse and pretty print JSON from HTTP responses. This will make tests easier to read and write and it will ensure hgweb is emitting well-formed JSON.

merge with stable

dirstate.walk: use the file foldmap to normalize Computing the set of directories in the dirstate is expensive. It turns out that it isn't necessary for operations like 'hg status' at all. Why? Consider the file 'foo/bar' on disk, which is represented in the dirstate as 'FOO/BAR'. On 'hg status', we'd walk down the directory tree, coming across 'foo' first. Before: we'd normalize 'foo' to 'FOO', then add 'FOO' to our visited stack. We'd then visit 'FOO', finding the file 'bar'. We'd normalize 'FOO/bar' to 'FOO/BAR', then add it to the results dict. After: we wouldn't normalize 'foo' at all. We'd add it to our visited stack, then visit 'foo', finding the file 'bar'. We'd normalize 'foo/bar' to 'FOO/BAR', then add it to the results dict. So whether we normalize intermediate directories or not actually makes no difference in most cases. The only case where normalization matters at all is if a file is replaced with a directory with the same case-folded name. In that case we can do a relatively cheap file normalization instead and still get away with not computing the set of directories. This is a nice boost in status performance. On OS X with case-insensitive HFS+, for a large repo with over 200,000 files, this brings down 'hg status' from 4.00 seconds to 3.62.

dirstate: split the foldmap into separate ones for files and directories Computing the set of directories in the dirstate can be pretty expensive. For 'hg status' without arguments, it turns out we actually never need to figure out the right case for directories in the foldmap. (An upcoming patch explains why.) This patch splits up the directory and file maps into separate ones, allowing for the subsequent optimization in status.

dirstate: introduce function to normalize just filenames This will be used in upcoming patches to stop generating the set of directories in many common cases.

dirstate: factor out code to discover normalized path In upcoming patches we're going to reuse this code. The storemap is currently always the foldmap, but will vary in future patches.

dirstate: don't require exact case when adding dirs on icasefs (issue4578) We don't require it when adding files on a case insensitive filesystem, so don't require it to add directories for consistency. The problem with the previous code was that _walkexplicit() was only returning the normalized directory. The file(s) in the directory are then appended, and passed to the matcher. But if the user asks for 'capsdir1/capsdir', the matcher will not accept 'CapsDir1/CapsDir/AbC.txt', and the name is dropped. Matching based on the non-normalized name is required. If not normalizing, skip the extra string building for efficiency. '.' is replaced with '' so that the path being tested when no file is specified, isn't prefixed with './' (and therefore fail the match).

filemerge: clean up language in mergemarkertemplate help

color: fix crash in cmd.exe When 'term' is None because it isn't in the environment, don't iterate over it. Unfortunately, unsetting $TERM or exporting it as '' doesn't work in the tests, so there's no way to simulate cmd.exe in the test suite.

log: prefer 'wctx' over 'pctx' for working context

merge with stable

tags: remove scary message about corrupt tags cache Caches should be transparent. If a cache is damaged, it should silently be rebuilt, much like if it were invalid. No one seems to have ever hit this in the wild.

hgk: use switch instead of a less efficient if/elseif/if

hgk: set distinct fill and outline colour for non-public and obsolete changesets

hgk: show secret changesets differently (shape and label)

manifestv2: implement slow readdelta() without revdiff For manifest v2, revlog.revdiff() usually does not provide enough information to produce a manifest. As a simple workaround, implement readdelta() by reading both the old and the new manifest and use manifest.diff() to find the difference. This is several times slower than the current readdelta() for v1 manifests, but there seems to be no other simple option, and this is still much faster than returning the full manifest (at least for verify).

manifestv2: disable fastdelta optimization We may add support for the fastdelta optimization for manifest v2 at a later point, but let's disable it for now, so we don't have to implement it right away.

manifestv2: add (unused) config option With tree manifests, hashes will change anyway, so now is a good time to also take up the old plans of a new manifest format. While there should be little or no reason to use tree manifests with the current manifest format (v1) once the new format (v2) is supported, we'll try to keep the two dimensions (flat/tree and v1/v2) separate. In preparation for adding a the new format, let's add configuration for it and propagate that configuration to the manifest revlog subclass. The new configuration ("experimental.manifestv2") says in what format to write the manifest data. We may later add other configuration to choose how to hash it, either keeping the v1 hash for BC or hashing the v2 content. See http://mercurial.selenic.com/wiki/ManifestV2Plan for more details.

manifest: extract method for creating manifest text Similar to the previous change, this one extracts a method for producing a manifest text from an iterator over (path, node, flags) tuples.

manifest: extract method for parsing manifest By extracting a method that generates (path, node, flags) tuples, we can reuse the code for parsing a manifest without doing it via a _lazymanifest like treemanifest currently does. It also prepares for parsing the new manifest format. Note that this makes parsing into treemanifest slower, since the parsing is now always done in pure Python. Since treemanifests will be expected (or even forced) to be used only with the new manifest format, parsing via _lazymanifest was not an option anyway.

dirstate._walkexplicit: don't bother normalizing '.' The overwhelmingly common case is running commands like 'hg diff' with no arguments. Therefore the only file that'll be listed is the root directory. Normalizing that's just a waste of time. This means that for a plain 'hg diff' we'll never need to construct the foldmap, saving us a significant chunk of time. On case-insensitive HFS+ on OS X, for a large repository with over 200,000 files, this brings down 'hg diff' from 2.97 seconds to 2.36.

dirstate._walkexplicit: drop normpath calls The paths the matcher returns are normalized already.

dirstate._walkexplicit: indicate root as '.', not '' '.' is the canonical way to represent the root, and it's apparently the only transformation that normpath makes.

phases: add killswitch for native implementation

phases: move pure phase computation in a function

revset: add hook after tree parsing This will be useful to execute actions after the tree is parsed and before the revset returns a match. Finding symbols in the parse tree will later allow hashes of hidden revisions to work on the command line without the --hidden flag.

hgk: remove unused revlog import

run-tests: obtain replacements inside Test._runcommand Now that command running is part of Test, we no longer need to pass a list of replacements down through various call layers. The impetus for this change is to fetch replacements after command execution, not before. This will allow replacements to be defined as part of test execution.

hgk: remove no longer needed debug-rev-parse command

hgk: remove no longer needed debug-config command

hgk: display obsolete changesets in darkgrey

hgk: pass --hidden switch to hg subprocesses when needed

hgk: remove repetitious (and wrong) command syntax descriptions

run-tests: separate newline normalization from replacements Upcoming patches will change how the replacements system works to make it more flexible. To prepare for this, eliminate the one-off use of replacements to perform newline normalization on Windows.

run-tests: remove arguments from Test._runcommand Now that runcommand is part of the Test class, arguments that were previously coming from Test attributes can now be switched to lookups inline.

run-tests: move run into Test class Future patches will change how replacements work. Since the logic in run() is strongly tied to the operation of individual tests and since there is potential to make the implementation simpler by giving the function access to Test attributes, move it into Test.

run-tests: wait for test threads after first error The test runner has the ability to stop on first error. Tests are executed in new Python threads. The test runner starts new threads when it has capacity to do so. Before this patch, the "stop on first error" logic would return immediately from the "run tests" function, without waiting on test threads to complete. There was thus a race between the test runner thread doing cleanup work and the test thread performing activity. For example, the test thread could be in the middle of executing a test shell script and the test runner could remove the test's temporary directory. Depending on timing, this could result in any number of output from the test runner. This patch eliminates the race condition by having the test runner explicitly wait for test threads to complete before continuing. I discovered this issue as I modified the test harness in a subsequent patch and was reliably able to tickle the race condition.

run-tests: report code coverage from source directory As part of testing code coverage output, I noticed some files were being reported twice: there was an entry for the file in the install location and for the file in the source tree. I'm not sure why this is. But it resulted in under-reporting of coverage data since some lines weren't getting covered in both locations. I also noticed that files in the source directory and outside the "mercurial" and "hgext" packages were getting included in the coverage report. Cosmetically, this seemed odd to me. It's not difficult to filter paths from the report. But I figure this data can be useful (we could start reporting run-tests.py coverage, for example). This patch switches the coverage API to report code coverage from the source directory. It registers a path alias so that data from the install location is merged into data from the source directory. We now get merged results for files that were being reported in multiple locations. Since code coverage reporting now relies on the profiled install now being in sync with the source tree, an additional check to disallow code coverage when --with-hg is specified has been added. This should have been present before, as --local was previously disallowed for the same reasons. Merging the paths raises our aggregate line coverage from ~60 to 81%.

run-tests: collect aggregate code coverage Before this patch, every Python process during a code coverage run was writing coverage data to the same file. I'm not sure if the coverage package even tries to obtain a lock on the file. But what I do know is there was some last write wins leading to loss of code coverage data, at least with -j > 1. This patch changes the code coverage mechanism to be multiple process safe. The mechanism for initializing code coverage via sitecustomize.py has been tweaked so each Python process will produce a separate coverage data file on disk. Unless two processes generate the same random value, there are no race conditions writing to the same file. At the end of the test run, we combine all written files into an aggregate report. On my machine, running the full test suite produces a little over 20,000 coverage files consuming ~350 MB. As you can imagine, it takes several seconds to load and merge these coverage files. But when it is done, you have an accurate picture of the aggregate code coverage for the entire test suite, which is ~60% line coverage.

run-tests: obtain code coverage via Python API Before, we were invoking the "coverage" program provided by the "coverage" module. This patch changes the code to go through the Python API. This makes the next patch a little bit easier to reason about. A side effect of this patch is that writing code coverage reports will be slightly faster, as we won't have to redundantly load coverage data.

commands.debugrevlog: report max chain length This is sometimes useful to know. Report it.

_lazymanifest: drop unnecessary call to sorted() The entries returned from _lazymanifest.iterentries() are already sorted.

test-git-export: add globs the test runner wants on Windows The only difference for the first two was to add the globs, but the third line of output on Windows was '..\dir2\copy'. I'm not sure why 'copy' is output on Windows instead of '*'.

run-tests: explicitly handle unicode when writing xunit file The xunit writer was passing a str to a minidom API. An implicit .decode('ascii') was performed somewhere, causing UnicodeDecodeError if test output contained non-ascii sequences. This patch converts test output to utf-8 before passing it to minidom. We use the "replace" strategy to ensure invalid utf-8 sequences get munged into �.

parsers.c: avoid implicit conversion loses integer warnings These warnings are raised by Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn) and were introduced in 539b3c7eea44

test-annotate: conditionalize error output for Windows It seems better to leave the actual output in place instead of globbing everything but 'abort:', in case it starts aborting for other reasons. It isn't clear the purpose for reversing the file name position, but that originates in windows.posixfile.

test-diffstat: add a glob the test runner wants on Windows The test gets a '~' status without it.

tests: add testing for diff.showfunc The diff.showfunc config knob did not have coverage before.

manifest: make manifest.intersectfiles() internal manifest.intersectfiles() is just a utility used by manifest.matches(), and a future commit removes intersectfiles for treemanifest for optimization purposes. This commit makes the intersectfiles methods on manifestdict and treemanifest internal, and converts its test to a more generic testMatches(), which has the exact same coverage.

win32: add comment about WinError Prevent reintroducing the bug that was added in e34106fa0dc3 (and fixed with a2285e2fc949).

templates: fix "log -q" output of phases style It had the same problem as 6136704b975d, name conflicts of {node} keyword.

record_curses: fix ui bug for newly added file With record's curses interface toggling and untoggling a newly added file would lead to a confusing UI (the header was marked as partial and the hunks as unselected). Tested additionally using the curses interface with newly added, removed and modified files in a test repo.

import-checker: rotatecycle is actually the canonical cycle key So refactor to drop cyclekey().