context: avoid breaking already fixed self._status at ctx.status() Before this patch, "status()" on "workingcommitctx" with "always match" object causes breaking "self._status" in "workingctx._buildstatus()", because "workingctx._buildstatus()" caches the result of "dirstate.status()" into "self._status" for efficiency, even though it should be fixed at construction for committing. For example, template function "diff()" without any patterns in "committemplate" implies "status()" on "workingcommitctx" with "always match" object, via "basectx.diff()" and "patch.diff()". Then, broken "self._status" causes committing unexpected files. To avoid breaking already fixed "self._status" at "ctx.status()", this patch overrides "_buildstatus" in "workingcommitctx". This patch doesn't write out the result of template function "diff()" in "committemplate" in "test-commit.t", because matching against files to be committed still has an issue fixed in subsequent patch.

context: add workingcommitctx for exact context to be committed Before this patch, "workingctx" is also used for the context to be committed. But "workingctx" works incorrectly in some cases. For example, even when only some of changed files in the working directory are committed, "status()" on "workingctx" object for committing recognizes files not to be committed as changed, too. As the preparation for fixing these issues, this patch chooses adding new class "workingcommitctx" for exact context to be committed, because switching by the flag (like "self._fixedstatus" or so) in some code paths of "workingctx" is less readable and maintenancable.

context: make unknown/ignored/clean of cached status empty for equivalence Before this patch, "workingctx.status" caches the result of "dirstate.status" directly into "self._status". But "dirstate.status" is invoked with False "list*" arguments in normal "self._status" accessing route, and this makes "unknown"/"ignored"/"clean" of status empty. This may cause unexpected result of code paths internally accessing to them (accessors for external usage are already removed by previous patch). This patch makes "unknown"/"ignored"/"clean" of cached status empty for equivalence. Making them empty is executed only when at least one of "unknown", "ignored" or "clean" has files, for efficiency.

templatefilters.json: stabilize output The json filter was previously iterating over keys in an object in an undefined order. Let's throw a sorted() in there so output is consistent. It's somewhat frightening that there are no tests for the json filter. Subsequent commits will add them, so we pass on the opportunity to add them here.

templatefilters.json: call functions The "changeset" template from hgweb is using a lambda in the "diffsummary" key. In preparation for enabling JSON output from hgweb, teach the json filter how to call functions.

merge with stable

linkrev: also adjust linkrev when bootstrapping annotate (issue4305) The annotate logic now use the new 'introrev' method to bootstrap its traversal. This catches issues from linkrev-shadowing of the changeset introducing the version of a file in source changeset. More tests have been added to display pathological cases.

linkrev: also adjust linkrev when bootstrapping 'follow' revset The follow revset (used by `hg log --follow`) now uses the new 'introrev' method to bootstrap its traversal. This catches issues from linkrev-shadowing of the changesets introducing the version of a file in source changeset. A new test has been added to display pathological cases. Another test is affected because it was meant to test this behavior but actually failed to do so for half of the output. The output are now similar.

linkrev: introduce an 'introrev' method on filectx The previous changeset properly fixed the ancestors computation, but we need to ensure that the initial filectx is also using the right changeset. When asking for log or annotation from a certain point, the first step is to define the changeset that introduced the current file version. We cannot just pick the "starting point" changesets as it may just "use" the file revision, unchanged. Currently, we were using 'linkrev' for this purpose, but this exposes us to unexpected branch-jumping when the revision introducing the starting point version is itself linkrev-shadowed. So we need to take the topology into account again. Therefore, we introduce an 'introrev' function, returning the changeset which introduced the file change in the current changeset. This function will be used to fix linkrev-related issues when bootstrapping 'hg log --follow' and 'hg annotate'. It reuses the '_adjustlinkrev' function, extending it to allow introspection of the initial changeset too. In the previous usage of the '_adjustlinkrev' the starting rev was always using a children file revisions, so it could be safely ignored in the search. In this case, the starting point is using the revision of the file we are looking, and may be the changeset we are looking for.

filectx.parents: enforce changeid of parent to be in own changectx ancestors Because of the way filenodes are computed, you can have multiple changesets "introducing" the same file revision. For example, in the changeset graph below, changeset 2 and 3 both change a file -to- and -from- the same content. o 3: content = new | | o 2: content = new |/ o 1: content = old In such cases, the file revision is create once, when 2 is added, and just reused for 3. So the file change in '3' (from "old" to "new)" has no linkrev pointing to it). We'll call this situation "linkrev-shadowing". As the linkrev is used for optimization purposes when walking a file history, the linkrev-shadowing results in an unexpected jump to another branch during such a walk.. This leads to multiple bugs with log, annotate and rename detection. One element to fix such bugs is to ensure that walking the file history sticks on the same topology as the changeset's history. For this purpose, we extend the logic in 'basefilectx.parents' so that it always defines the proper changeset to associate the parent file revision with. This "proper" changeset has to be an ancestor of the changeset associated with the child file revision. This logic is performed in the '_adjustlinkrev' function. This function is given the starting changeset and all the information regarding the parent file revision. If the linkrev for the file revision is an ancestor of the starting changeset, the linkrev is valid and will be used. If it is not, we detected a topological jump caused by linkrev shadowing, we are going to walk the ancestors of the starting changeset until we find one setting the file to the revision we are trying to create. The performance impact appears acceptable: - We are walking the changelog once for each filelog traversal (as there should be no overlap between searches), - changelog traversal itself is fairly cheap, compared to what is likely going to be perform on the result on the filelog traversal, - We only touch the manifest for ancestors touching the file, And such changesets are likely to be the one introducing the file. (except in pathological cases involving merge), - We use manifest diff instead of full manifest unpacking to check manifest content, so it does not involve applying multiple diffs in most case. - linkrev shadowing is not the common case. Tests for fixed issues in log, annotate and rename detection have been added. But this changeset does not solve all problems. It fixes -ancestry- computation, but if the linkrev-shadowed changesets is the starting one, we'll still get things wrong. We'll have to fix the bootstrapping of such operations in a later changeset. Also, the usage of `hg log FILE` without --follow still has issues with linkrev pointing to hidden changesets, because it relies on the `filelog` revset which implement its own traversal logic that is still to be fixed. Thanks goes to: - Matt Mackall: for nudging me in the right direction - Julien Cristau and Rémi Cardona: for keep telling me linkrev bug were an evolution show stopper for 3 years. - Durham Goode: for finding a new linkrev issue every few weeks - Mads Kiilerich: for that last rename bug who raise this topic over my anoyance limit.

context: remove unreliable accessor methods from committablectx There are two caching routes for (propertycache-ed) "_status" below in committablectx: - invoking "status()": "dirstate.status()" is invoked, and the result of it is cached into "_status". In this case, any of "listignored", "listclean" and "listunknown" may be True. - accessing "_status" directly before "status()": Own "status()" is invoked, but all of "listignored", "listclean" and "listunknown" arguments are False, in this case. "ignored"/"clean"/"unknown" accessor methods of "committablectx" use corresponded fields of "_status", but these fields aren't reliable, because these fields are empty when: - "_status" method is executed before accessors, or - "status()" is executed with "list*=False" before accessors In addition to it, these accessors aren't used in the recent Mercurial implementation. At least, removing them doesn't cause any test failures.

context: cache self._status correctly at workingctx.status Before this patch, "workingctx.status" always replaces "self._status" by the recent result, even though: - status isn't calculated against the parent of the working directory, or - specified "match" isn't "always" one (status is only visible partially) If "workingctx" object is shared between some procedures indirectly referring "ctx._status", this incorrect caching may cause unexpected result: for example, "ctx._status" is used via "manifest()", "files()" and so on. To cache "self._status" correctly at "workingctx.status", this patch overwrites "self._status" in "workingctx._buildstatus" only when: - status is calculated against the parent of the working directory, and - specified "match" is "always" one This patch can be applied (and effective) only on default branch, because procedure around "basectx.status" is much different between stable and default: for example, overwriting "self._status" itself is executed not in "workingctx._buildstatus" but in "workingctx._poststatus", on stable branch.