revert: take fast path also when not reverting to '.' This speeds up 'hg revert -r .^ --all --dry-run' on the Mozilla repo from 4.081s to 0.826s. Note that 'hg revert -r .^ .' does not get any faster, since '.' does not make match.always() True. I can't think of a reason it would break anything, and if it does, it's clearly not covered by tests.

revert: define 'wctx' a little earlier and use it more

match: add isexact() method to hide internals Comparing a function reference seems bad.

matcher: make e.g. 'relpath:.' lead to fast paths Several commands take the fast path when match.always() is true. However, when the user passes "." on the command line, that results in a matcher for which match.always() == False. Let's make it so such matchers return True, and have an empty list of .files(). This makes e.g. "hg log ." as fast as "hg log" and "hg revert ." as fast as "hg revert --all" (when run from repo root).

revset: add the 'subrepo' symbol This returns the csets where matching subrepos have changed with respect to the containing repo's first parent. The second parent shouldn't matter, because it is either syncing up to the first parent (i.e. it hasn't changed from the current branch's POV), or the merge changed it with respect to the first parent (which already adds it to the set). There's already a 'subrepo' fileset, but it is prefixed with 'set:', so there should be no ambiguity (in code anyway). The only test I see for it is to revert subrepos named by a glob pattern (in test-subrepo.t, line 58). Since it doesn't return a tracked file, neither 'log "set:subrepo()"' nor 'files "set:subrepo()"' print anything. Therefore, it seems useful to have a revset that will return something for log (and can be added to a revsetalias to be chained with 'file' revsets.) It might be nice to be able to filter for added, modified and removed separately, but add/remove should be rare. It might also be nice to be able to do a 'contains' check, in addition to this mutated check. Maybe it is possible to get those with the existing 'adds', 'contains', 'modifies' and 'removes' by teaching them to chase explicit paths into subrepos. I'm not sure if this should be added to the 'modifies adds removes' line in revset.optimize() (since it is doing an AMR check on .hgsubstate), or if it is OK to put into 'safesymbols' (things like 'file' are on the list, and that takes a regex, among other patterns).

tags: improve documentation The documentation for tags.py was making comprehension difficult. This patch rewrites most of the comments in the file to make it easier for mere mortals to understand what's going on.

phase: default to C implementation for phase computation

phase: compute phases in C Previously, the phase computation would grow much slower as the oldest draft commit in the repository grew older (which is very common in repos with evolve on) and the number of commits increase. By rewriting the computation in C we can speed it up from 700ms to 7ms on a large repository whose oldest draft commit is a year old.

manifest: move C bool polyfill into util.h

manifest: use util.h to get Py_ssize_t

clone: add progress support to hardlink clones (issue3059)

util: add progress callback support to copyfiles

revert: evaluate filesets against working directory (issue4497) As the failing revert tests in test-fileset-generated.t show, Revert currently creates one matcher for matching files in the working copy and another matcher for matching files in the target revision. The two matchers are created with different contexts, which means filesets are evaluated differently. Then the union of the sets of files matching the matchers in the two contexts are reverted. It doesn't seem to make sense to use two different matchers; only the context they're applied to should be different. It seems very likely that the user wants the filesets to be evaluated against the working directory, which the tests test-fileset-generated.t also assume, so let's make it so. I willingly admit that the largefiles code was modified by trial and error (according to tests).

largefiles: extract and reuse 'standin' variable in overriderevert()

largefiles: override cmdutil.revert() instead of comands.revert() By overriding the cmdutil method we don't need to override both the function and the command. Also, we get access to the 'ctx' and 'parents' variables, which will soon prove useful. Rename the 'ctx' argument to overridematch() to 'mctx' rather than letting it shadow new 'ctx'.

test-commit-interactive-curses: backout 852801178a63 Now that the imports have been fixed up, and the platform EOL difference eliminated, this test runs on Windows.

test-interactive: use stable EOL in various file generating routines The seq.py and shell echo redirected to a file ends up with platform specific EOL, which throws off the hash when the file is committed to the test repo on Windows. The other option is to glob the hashes, but the ability to see if they have changed might point out problems that would otherwise be missed.

patch.diff: restrict matcher to relative root in certain cases Previously we'd request all results, then filter by relative root. This is clearly inefficient, so we now restrict the matcher to the relative root for certain easy cases. The particular case here is when the matcher matches all files. In that case we can simply create a matcher by the relative root. This is purely an optimization and has no impact on correctness.

commands.diff: add support for diffs relative to a subdirectory Previous patches added all the backend support for this. This exposes this option in the UI.

cmdutil.diffordiffstat: add support for diffs relative to a subdirectory It's possible that some of the match patterns are outside the relative root. We warn in that case. In upcoming patches we'll add tests for this.

test-fileset.t: remove 'bar ' and 'baz\' from tested filenames This test is guarded by a '#if no-windows', but they can't be written out on vfat on Linux either. con.xml can, and for testing that's good enough. Also clean up a stray 'ls' that I mistakenly left in while writing the test.

commands.push: abort when revisions evaluate to empty set (BC) If the "-r" argument is specified to "hg push," the user has expressed an intent for a specific changeset to be present on the remote. If that expression cannot be mapped to a known changeset, the user's intent is ambiguous and cannot be acted upon without making assumptions. Previously, if arguments to `push -r <rev>` evaluated to an empty set (perhaps the user specified a revset that didn't evaluate to anything), the empty "revs" list would be passed down to "exchange.push" where it appears the empty list was being interpreted as "push everything." This patch adds validation to the "-r" argument to the push command. If the argument is specified but doesn't resolve to a changeset, the command will abort instead of doing something potentially unexpected. This patch is technically breaking backwards compatibility. I believe this is justified because the new behavior closes a crack that could result in undefined or under-defined behavior. Also, this patch doesn't drop client capabilities because if users really wanted to push all changesets, they can simply omit the "-r" argument from push completely.

largefiles: fix typo with s/__class/__class__/ Commit f78252429e0a (largefiles: don't create chain of __contains__ calls, 2015-03-11) introduced a typo with __class instead of __class__. Let's fix it. There are (obviously) no tests covering this code path, and I could not figure out a way to trigger it, so it remains untested. Spotted by Drew Gottlieb.

hgweb: pull line numbers out of main flow of source content Pulls the autogenerated line number boxes outside of the padding box of the main content of .sourcelines, allowing the first tab to be properly sized and the line numbers to be outside the main source's margin when text is wrapped.

merge with stable

crecord: don't abort on Windows when curses isn't available This is the hack portion of the previous patch, which can be backed out once we figure out how to deal with curses. Without this, the tests affected by the import problem mentioned in the previous patch fail in exactly the same way, but with the nicer 'module not available' error.

test-fileset: forget non-portable files to avoid failures on Windows The 3 deleted lines appeared on Windows, because the files were never created in the first place.

crecord: conditionalize the imports that are not available on Windows 'fcntl', 'termios' and 'wcurses' are not available on the default Windows python installation, and importing them caused widespread carnage in the test suite. There were 29 different changed test files (on top of unrelated errors), mostly in the form of an ImportError. The failures weren't related to actual crecord use, and followed the import chain: 'localrepo' -> 'subrepo' -> 'cmdutil' -> 'crecord' -> 'fcntl'

cmdutil: drop a duplicate import of crecord It looks like this crept in with 69538481ea9f.

annotate: add option to annotate working-directory files Working revision or node is displayed with "+" suffix in plain output, but null/None in machine-readable format.

committablefilectx: override linkrev() to point to the associated changectx This is necessary to annotate workingctx revision. basefilectx.linkrev() can't be used because committablefilectx has no filelog. committablefilectx looks for parents() from self._changectx. That means fctx is linked to self._changectx, so linkrev() can simply be aliased to rev().

revset: add wdir() function to specify workingctx revision by command The main purpose of wdir() is to annotate working-directory files. Currently many commands and revsets cannot handle workingctx and may raise exception. For example, -r ":wdir()" results in TypeError. This problem will be addressed by future patches. We could add "wdir" symbol instead, but it would conflict with the existing tag, bookmark or branch. So I decided not to. List of commands that will potentially support workingctx revision: command default remarks -------- ------- ----------------------------------------------------- annotate p1 useful archive p1 might be useful cat p1 might be useful on Windows (no cat) diff p1:wdir (default) export p1 might be useful if wctx can have draft commit message files wdir (default) grep tip:0 might be useful identify wdir (default) locate wdir (default) log tip:0 might be useful with -p or -G option parents wdir (default) status wdir (default) This patch includes minimal test of "hg status" that should be able to handle the workingctx revision.

win32: 'raise ctypes.WinError' -> 'raise ctypes.WinError()' WinError is a function that creates an Error, not an Error itself. This is a partial backout of e34106fa0dc3.

patch.diff: add support for diffs relative to a subdirectory For now this implementation is pretty naive -- it filters out files right before passing them into trydiff. In upcoming patches we'll add some more smarts.

patch.trydiff: add support for stripping a relative root This assumes that if relroot is not None, all the files in modified, added and removed start with it. In upcoming patches we'll follow that.

merge with stable

obsolete: mark warning for translation No good reason not to.

subrepo: add basic support to hgsubrepo for the files command Paths into the subrepo are not yet supported. The need to use the workingctx in the subrepo will likely be used more in the future, with the proposed working directory revset symbol. It is also needed with archive, if that code is to be reused to support 'extdiff -S'. Unfortunately, it doesn't seem possible to put the smarts in subrepo.subrepo(), as it breaks various status and diff tests. I opted not to pass the desired revision into the subrepo method explicitly, because the only ones that do pass an explicit revision are methods like status and diff, which actually operate on two contexts- the subrepo state and the explicitly passed revision.

mergecopies: reuse ancestry context when traversing file history (issue4537) Merge copies is traversing file history in search for copies and renames. Since 3.3 we are doing "linkrev adjustment" to ensure duplicated filelog entry does not confuse the traversal. This "linkrev adjustment" involved ancestry testing and walking in the changeset graph. If we do such walk in the changesets graph for each file, we end up with a 'O(<changesets>x<files>)' complexity that create massive issue. For examples, grafting a changeset in Mozilla's repo moved from 6 seconds to more than 3 minutes. There is a mechanism to reuse such ancestors computation between all files. But it has to be manually set up in situation were it make sense to take such shortcut. This changesets set this mechanism up and bring back the graph time from 3 minutes to 8 seconds. To do so, we need a bigger control on the way 'filectx' are instantiated during each 'checkcopies' calls that 'mergecopies' is doing. We add a new 'setupctx' that configure and return a 'filectx' factory. The function make sure the ancestry context is properly created and the factory make sure it is properly installed on returned 'filectx'.

adjustlinkrev: handle 'None' value as source When the source rev value is 'None', the ctx is a working context. We cannot compute the ancestors from there so we directly skip to its parents. This will be necessary to allow 'None' value for '_descendantrev' itself necessary to make all contexts used in 'mergecopies' reuse the same '_ancestrycontext'.

adjustlinkrev: prepare source revs for ancestry only once We'll need some more complex initialisation to handle workingfilectx case. We do this small change in a different patch for clarity.

subrepo: add the parent context to hgsubrepo This brings parity with gitsubrepo and svnsubrepo (which already reference their parent), and will be used in an upcoming patch. I'm a bit concerned that the parent context could get stale (consider what happens when the parent repo is reverted for example). I tried adding the parent context to the substate tuple so that the parent is available everywhere a state change is possible, but that made submerge() unhappy. Even with removing the parent context inside submerge(), I wasn't able to get all of the test diffs fixed. But since the other subrepos reference their parent too, if there is a problem, it is a preexisting one (that nobody seems to be running into). It can be fixed if/when it pops up.

fileset: add a fileset for portable filenames This has mostly the same semantics as the files that the 'ui.portablefilenames' config option would warn or abort about. The only difference is filenames that case-fold to the same string -- given a set of filenames we've already checked we can check whether a new one collides with them, but we don't have a way to tell which filename it collided with.

annotate: reuse ancestry context when adjusting linkrev (issue4532) The linkrev adjustment will likely do the same ancestry walking multiple time so we already have an optional mechanism to take advantage of this. Since 2896f53509a7, linkrev adjustment was done lazily to prevent too bad performance impact on rename computation. However, this laziness created a quadratic situation in 'annotate'. Mercurial repo: hg annotate mercurial/commands.py before: 8.090 after: 36.300 Mozilla repo: hg annotate layout/generic/nsTextFrame.cpp before: 1.190 after: 290.230 So we setup sharing of the ancestry context in the annotate case too. Linkrev adjustment still have an impact but it a much more sensible one. Mercurial repo: hg annotate mercurial/commands.py before: 36.300 after: 10.230 Mozilla repo: hg annotate layout/generic/nsTextFrame.cpp before: 290.230 after: 5.560

treemanifest: make hasdir() faster Same rationale as the previous change.

treemanifest: make filesnotin() faster Same rationale as the previous change.

treemanifest: make diff() faster Containment checking is slower in treemanifest than it is in manifestdict, making the current diff algorithm O(n log n). By traversing both treemanifests in parallel, we can make it O(n). More importantly, once we start lazily loading submanifests, we will be able to easily skip entire submanifest if they have the same nodeid.

treemanifest: store directory path in treemanifest nodes This leads to less concatenation while iterating, and it's useful for debugging.

treemanifest: add configuration for using treemanifest type This change adds boolean configuration option experimental.treemanifest. When the option is enabled, manifests are parsed into the new treemanifest type. Tests can be now run using treemanifest by switching the config option default in localrepo._applyrequirements(). Tests pass even when made to randomly choose between manifestdict and treemanifest, suggesting that the two types produce identical manifests (so e.g. a manifest revlog entry written from a treemanifest can be parsed by the manifestdict code).

treemanifest: create treemanifest class There are a number of problems with large and flat manifests. Copying from http://mercurial.selenic.com/wiki/ManifestShardingPlan: * manifest too large for RAM * manifest resolution too much CPU (long delta chains) * committing is slow because entire manifest has to be hashed * impossible for narrow clone to leave out part of manifest as all is needed to calculate new hash * diffing two revisions involves traversing entire subdirectories even if identical This is a first step in a series introducing a manifest revlog per directory. This change adds a new manifest class: treemanifest, which is a tree where each node has a dict of files (nodeids), a dict of flags, and a dict of subdirectories (treemanifests). So far, it behaves just like manifestdict, but it will later help us write one manifest revlog per directory. The new class is still unused; it will be used after the next change. The code is not yet optimized. Running with it (see below) makes most or all operations slower. Once we start storing manifest revlogs for every directory, it should be possible to make many of these operations much faster. The fastdelta() optimization has been intentionally not implemented for the treemanifests. We can implement it later if necessary. All tests pass when run with the following patch (and without, of couse): --- a/mercurial/manifest.py Thu Mar 19 11:08:42 2015 -0700 +++ b/mercurial/manifest.py Thu Mar 19 11:15:50 2015 -0700 @@ -596,7 +596,7 @@ class manifest(revlog.revlog): return None, None def add(self, m, transaction, link, p1, p2, added, removed): - if p1 in self._mancache: + if False and p1 in self._mancache: # If our first parent is in the manifest cache, we can # compute a delta here using properties we know about the # manifest up-front, which may save time later for the @@ -626,3 +626,5 @@ class manifest(revlog.revlog): self._mancache[n] = (m, arraytext) return n + +manifestdict = treemanifest

bookmarks: rewrite comparing bookmarks in commands.summary() by compare() This patch adds utility function "summary()", to replace comparing bookmarks in "commands.summary()". This replacement finishes centralizing the logic to compare bookmarks into "bookmarks.compare()". This patch also adds test to check summary output with incoming/outgoing bookmarks, because "hg summary --remote" is not tested yet on the repository with incoming/outgoing bookmarks. This test uses "(glob)" to ignore summary about incoming/outgoing changesets.

bookmarks: remove useless diff() Previous patches removed code paths referring it.

bookmarks: add outgoing() to replace diff() for outgoing bookmarks This replacement makes enhancement of "show outgoing bookmarks" easy, because "compare()" can detect more detailed difference of bookmarks between two repositories.

bookmarks: add incoming() to replace diff() for incoming bookmarks This replacement makes enhancement of "show incoming bookmarks" easy, because "compare()" can detect more detailed difference of bookmarks between two repositories.

manifest: avoid intersectfiles for matches > 100 files Previously we tried to avoid manifest.intersectfiles for exact matches with less than 100 files. However, when the left side of the "or" is false, the right side gets evaluated, of course, and the evaluation of "util.all(fn in self for fn in files)" is both costly in itself, and likely to be true, causing intersectfiles() to be called after all. Fix this by moving the check for less than 100 files outside of the "or" expression, thereby also making it apply for a non-exact matcher, should one be passed in.

convert: optimize convert of files that are unmodified from p2 in merges Conversion of a merge starts with p1 and re-adds the files that were changed in the merge or came unmodified from p2. Files that are unmodified from p1 will thus not be touched and take no time. Files that are unmodified from p2 would be retrieved and rehashed. They would end up getting the same hash as in p2 and end up reusing the filelog entry and look like the p1 case ... but it was slow. Instead, make getchanges also return 'files that are unmodified from p2' so the sink can reuse the existing p2 entry instead of calling getfile. Reuse of filelog entries can make a big difference when files are big and with long revlong chains so they take time to retrieve and hash, or when using an expensive custom getfile function (think http://mercurial.selenic.com/wiki/ConvertExtension#Customization with a code reformatter). This in combination with changes to reuse filectx entries in localrepo._filecommit make 'unchanged from p2' almost as fast as 'unchanged from p1'. This is so far only implemented for the combination of hg source and hg sink. This is a refactoring/optimization. It is covered by existing tests and show no changes - which is a good thing.

localrepo: reuse commit of parent filectx entries without rehashing It is currently only amend and debugbuilddag that will pass a filectx to localrepo._filecommit. Amend will usually not hit the case where a the filectx is a parent that just can be reused. Future convert changes will use it more.

obsolete: avoid infinite loop from obs-cycle in divergence (issue4126) As for other currently in place cycle detection, arbitrarily cut the first obsolescence link that create a cycle avoiding the infinite loop. This will have to be made more deterministic in the future but we do not really care right now.

tests: avoid deprecation warning

walkchangerevs: make followfilter a top-level class The class only depends on the 'repo' variable in the closure, so let's move the class out of the function and make it explicit that that (the repo) is all it needs.