merge with stable

cmdutil.jsonchangeset: properly compute added and removed files jsonchangeset._show() was computing the reverse status of the current changeset. As a result, added files were showing up as removed and removed files were showing up as adds. There were existing tests for this code and they were flat out wrong.

largefiles: convert addlargefiles() to vfs

rebase: clarify comment about merge ancestor when rebasing merges The code for picking a merge ancestor when rebasing merges had a long and incorrect comment. The comment would perhaps have been fine as commit message but does not make the code more readable or maintainable and is a bad substitute for correct and readable code. The correct essense of the comment is quite trivial: a merge of an ancestor of the rebase destination and an 'outside' revision can be rebased as if it was a linear change, using 'destination ancestor parent' as base and pretty much ignoring the 'outside' revision. The code path where the comment is placed is however also used for other kinds of merge rebases. The comment is thus not really correct and not helpful. I think it would be better to drop the comment and rewrite the code.

status: don't list files as both clean and deleted Tracked files that are deleted should always be reported as such, no matter what their state was in earlier revisions. This is encoded in in two conditions in the loop in basectx._buildstatus() for modified and added files, but the check is missing for clean files. We should check for clean files too, but instead of adding the check in a third place, move it earlier and skip most of the loop body for deleted files.

status: don't list files as both removed and deleted When calculating status involving the working copy and a revision other than the parent of the working copy, the files that are not in the working context manifest ('mf2' in the basectx._buildstatus()) will be reported as removed (note that deleted files _are_ in the working context manifest). However, if the file is reported as deleted in the dirstate, it will get that status too (as shown by failing tests). Fix by removing deleted files from the 'removed' list after the main loop in _buildstatus().

revset-filelog: handle hidden linkrev for file missing for head (issue4490) The fix for linkrev pointing to hidden revision was crashing when the file was missing from head's manifest. We now properly handle this case. (yes I feel silly)

run-tests: automatically add (glob) to "saved backup bundle to" lines Avoid spending too much time adding (glob) after running run-tests -i. This doesn't handle all cases but it helps. The run-tests tests add a bit of escaping of trailing (glob) in the output to avoid interference from the outer test runner. The regexp for matching the output lines contains a group for making multiline substitute in a way that works with Python before 2.7.

transaction: use the right location when cleaning up backup file (issue4479) The location variable fetch from the loop and the one used to actually fetch it mismatched. We fix the name to ensure file outside of store are cleaned up.

merge with stable

revset: delay showing parse error for the revset alias until it is referred Before this patch, a problematic revset alias aborts execution immediately, even if it isn't referred in the specified revset. If old "hg" may be used too (for example, bisecting Mercurial itself), it is also difficult to write alias definitions using features newly introduced by newer "hg" into configuration files, because such alias definitions cause unexpected abortion at parsing revset aliases with old "hg". This patch delays showing parse error for the revset alias until it is actually referred at runtime. This patch detects referring problematic aliases in "_expandaliases" by examination of "revsetalias.error", which is initialized with the error message only when parsing fails. For usability, this patch also warns about problematic aliases, even if they aren't referred at runtime. This should help users to know potential problems in their alias definitions earlier.

revset: drop pre-lazyset optimization for stringset of subset == entire repo It was introduced at e44ebd2a142a, where spanset.__contains__() did not exist. Nowadays, we have to pay huge penalty for len(subset). The following example showed that OR operation could be O(n * m^2) (n: len(repo), m: number of OR operators, m >= 2) probably because of filteredset.__len__. revset #0: 0|1|2|3|4|5|6|7|8|9 0) wall 8.092713 comb 8.090000 user 8.090000 sys 0.000000 (best of 3) 1) wall 0.445354 comb 0.450000 user 0.430000 sys 0.020000 (best of 22) 2) wall 0.000389 comb 0.000000 user 0.000000 sys 0.000000 (best of 7347) (0: 3.2.4, 1: 3.1.2, 2: this patch)

keyword: update copyright year

keyword: use vfs.reljoin and util.unlinkpath to remove kwdemo

largefiles: properly sync lfdirstate after removing largefiles The more aggressive synchronization of lfdirstate that was backed out in 1265a3a71d75 masked the problem where lfdirstate would hold an 'R' for a largefile that was added and then removed without a commit between. We could just conditionally call lfdirstate.drop() or lfdirstate.remove() here, but this also properly updates lfdirstate if the standin doesn't exist for the file somehow (i.e. call drop instead of remove). Without this change, the precommit status in the commit command immediately after the test change lists the removed (and never committed) largefile as 'R'. It can also lead to situations where the status command reports the same, long after the commit [1]. [1] http://www.selenic.com/pipermail/mercurial-devel/2015-January/065153.html

linkrev-filelog: handle filtered linkrev with no visible children (issue4307) If the file revision with a filtered linkrev does not have any (unfiltered) children, we cannot use it to bound the search for another introduction. Instead, we have to look at the file revision used by each head changeset. If one of them uses this file revision, we know there is another occurrence and we have a starting point. See inline comments for details. Adding some kind of permanent reference of all the introductions of a file revision instead of just the first one would be much better. But this is more difficult. I hope to take that into account in the next repository format.

linkrev: work around linkrev to filtered entry in 'filelog' revset This revset is used by 'hg log FILENAME'. This prevent bugs when used on a repository with hidden revisions. Instead of just discarding file revisions whose linkrevs point to filtered revisions, we put them aside and post-process them trying to find a non-filtered introduction. See inline documentation for details about how it works. This only fixes some of the problems. Once again, more will be needed when we can cannot rely on child revisions of a file to find linkrev-shadowned revisions. A test is added for 'hg log' catching such cases.

namespaces: update documentation and code indentation The previous patch changed just the functionality, while this patch focuses on the documentation and indentation to keep review simple.

namespaces: use namespace object instead of dictionary This isn't as bad as the diff seems, it only looks like scary. In this patch, we use the 'namespace' object instead of accessing keys in a dictionary. This required the 'templatename' and 'names' method to change their implementation. Later, we will remove these functions entirely due to a better api.

namespaces: copy implementation to new namespace object

namespaces: add a namespace object Currently, we use a dictionary object to store the namespace properties. This is python so let's use an object. This will allow us to be more flexible in our method signatures in the future.

localrepo: use vfs.reljoin rather than os.path.join in the localrepository class The removes the last remaining usage of os.path.join inside teh localrepository class.

largefiles: drop a redundant check for patterns when removing without --after This is in the way of passing a matcher to removelargefiles(). This method is called in exactly two places- first in overrides.addremove() (but only if the pattern list passed to it is not empty), and second in the commands.remove() override. But since the latter calls commands.remove() first, which also does this check, it isn't needed here.

context: override _dirstatestatus in workingcommitctx for correct matching Before this patch, the result of "status()" on "workingcommitctx" may incorrectly contain files other than ones to be committed, because "workingctx._dirstatestatus()" returns the result of "dirstate.status()" directly. For correct matching, this patch overrides "_dirstatestatus" in "workingcommitctx" and makes it return matched files only in "self._status". This patch uses empty list for "deleted", "unknown" and "ignored" of status, because status between "changectx"s also makes them empty.

context: avoid breaking already fixed self._status at ctx.status() Before this patch, "status()" on "workingcommitctx" with "always match" object causes breaking "self._status" in "workingctx._buildstatus()", because "workingctx._buildstatus()" caches the result of "dirstate.status()" into "self._status" for efficiency, even though it should be fixed at construction for committing. For example, template function "diff()" without any patterns in "committemplate" implies "status()" on "workingcommitctx" with "always match" object, via "basectx.diff()" and "patch.diff()". Then, broken "self._status" causes committing unexpected files. To avoid breaking already fixed "self._status" at "ctx.status()", this patch overrides "_buildstatus" in "workingcommitctx". This patch doesn't write out the result of template function "diff()" in "committemplate" in "test-commit.t", because matching against files to be committed still has an issue fixed in subsequent patch.

context: add workingcommitctx for exact context to be committed Before this patch, "workingctx" is also used for the context to be committed. But "workingctx" works incorrectly in some cases. For example, even when only some of changed files in the working directory are committed, "status()" on "workingctx" object for committing recognizes files not to be committed as changed, too. As the preparation for fixing these issues, this patch chooses adding new class "workingcommitctx" for exact context to be committed, because switching by the flag (like "self._fixedstatus" or so) in some code paths of "workingctx" is less readable and maintenancable.

context: make unknown/ignored/clean of cached status empty for equivalence Before this patch, "workingctx.status" caches the result of "dirstate.status" directly into "self._status". But "dirstate.status" is invoked with False "list*" arguments in normal "self._status" accessing route, and this makes "unknown"/"ignored"/"clean" of status empty. This may cause unexpected result of code paths internally accessing to them (accessors for external usage are already removed by previous patch). This patch makes "unknown"/"ignored"/"clean" of cached status empty for equivalence. Making them empty is executed only when at least one of "unknown", "ignored" or "clean" has files, for efficiency.

templatefilters.json: stabilize output The json filter was previously iterating over keys in an object in an undefined order. Let's throw a sorted() in there so output is consistent. It's somewhat frightening that there are no tests for the json filter. Subsequent commits will add them, so we pass on the opportunity to add them here.

templatefilters.json: call functions The "changeset" template from hgweb is using a lambda in the "diffsummary" key. In preparation for enabling JSON output from hgweb, teach the json filter how to call functions.

merge with stable

linkrev: also adjust linkrev when bootstrapping annotate (issue4305) The annotate logic now use the new 'introrev' method to bootstrap its traversal. This catches issues from linkrev-shadowing of the changeset introducing the version of a file in source changeset. More tests have been added to display pathological cases.

linkrev: also adjust linkrev when bootstrapping 'follow' revset The follow revset (used by `hg log --follow`) now uses the new 'introrev' method to bootstrap its traversal. This catches issues from linkrev-shadowing of the changesets introducing the version of a file in source changeset. A new test has been added to display pathological cases. Another test is affected because it was meant to test this behavior but actually failed to do so for half of the output. The output are now similar.

linkrev: introduce an 'introrev' method on filectx The previous changeset properly fixed the ancestors computation, but we need to ensure that the initial filectx is also using the right changeset. When asking for log or annotation from a certain point, the first step is to define the changeset that introduced the current file version. We cannot just pick the "starting point" changesets as it may just "use" the file revision, unchanged. Currently, we were using 'linkrev' for this purpose, but this exposes us to unexpected branch-jumping when the revision introducing the starting point version is itself linkrev-shadowed. So we need to take the topology into account again. Therefore, we introduce an 'introrev' function, returning the changeset which introduced the file change in the current changeset. This function will be used to fix linkrev-related issues when bootstrapping 'hg log --follow' and 'hg annotate'. It reuses the '_adjustlinkrev' function, extending it to allow introspection of the initial changeset too. In the previous usage of the '_adjustlinkrev' the starting rev was always using a children file revisions, so it could be safely ignored in the search. In this case, the starting point is using the revision of the file we are looking, and may be the changeset we are looking for.

filectx.parents: enforce changeid of parent to be in own changectx ancestors Because of the way filenodes are computed, you can have multiple changesets "introducing" the same file revision. For example, in the changeset graph below, changeset 2 and 3 both change a file -to- and -from- the same content. o 3: content = new | | o 2: content = new |/ o 1: content = old In such cases, the file revision is create once, when 2 is added, and just reused for 3. So the file change in '3' (from "old" to "new)" has no linkrev pointing to it). We'll call this situation "linkrev-shadowing". As the linkrev is used for optimization purposes when walking a file history, the linkrev-shadowing results in an unexpected jump to another branch during such a walk.. This leads to multiple bugs with log, annotate and rename detection. One element to fix such bugs is to ensure that walking the file history sticks on the same topology as the changeset's history. For this purpose, we extend the logic in 'basefilectx.parents' so that it always defines the proper changeset to associate the parent file revision with. This "proper" changeset has to be an ancestor of the changeset associated with the child file revision. This logic is performed in the '_adjustlinkrev' function. This function is given the starting changeset and all the information regarding the parent file revision. If the linkrev for the file revision is an ancestor of the starting changeset, the linkrev is valid and will be used. If it is not, we detected a topological jump caused by linkrev shadowing, we are going to walk the ancestors of the starting changeset until we find one setting the file to the revision we are trying to create. The performance impact appears acceptable: - We are walking the changelog once for each filelog traversal (as there should be no overlap between searches), - changelog traversal itself is fairly cheap, compared to what is likely going to be perform on the result on the filelog traversal, - We only touch the manifest for ancestors touching the file, And such changesets are likely to be the one introducing the file. (except in pathological cases involving merge), - We use manifest diff instead of full manifest unpacking to check manifest content, so it does not involve applying multiple diffs in most case. - linkrev shadowing is not the common case. Tests for fixed issues in log, annotate and rename detection have been added. But this changeset does not solve all problems. It fixes -ancestry- computation, but if the linkrev-shadowed changesets is the starting one, we'll still get things wrong. We'll have to fix the bootstrapping of such operations in a later changeset. Also, the usage of `hg log FILE` without --follow still has issues with linkrev pointing to hidden changesets, because it relies on the `filelog` revset which implement its own traversal logic that is still to be fixed. Thanks goes to: - Matt Mackall: for nudging me in the right direction - Julien Cristau and Rémi Cardona: for keep telling me linkrev bug were an evolution show stopper for 3 years. - Durham Goode: for finding a new linkrev issue every few weeks - Mads Kiilerich: for that last rename bug who raise this topic over my anoyance limit.

context: remove unreliable accessor methods from committablectx There are two caching routes for (propertycache-ed) "_status" below in committablectx: - invoking "status()": "dirstate.status()" is invoked, and the result of it is cached into "_status". In this case, any of "listignored", "listclean" and "listunknown" may be True. - accessing "_status" directly before "status()": Own "status()" is invoked, but all of "listignored", "listclean" and "listunknown" arguments are False, in this case. "ignored"/"clean"/"unknown" accessor methods of "committablectx" use corresponded fields of "_status", but these fields aren't reliable, because these fields are empty when: - "_status" method is executed before accessors, or - "status()" is executed with "list*=False" before accessors In addition to it, these accessors aren't used in the recent Mercurial implementation. At least, removing them doesn't cause any test failures.

context: cache self._status correctly at workingctx.status Before this patch, "workingctx.status" always replaces "self._status" by the recent result, even though: - status isn't calculated against the parent of the working directory, or - specified "match" isn't "always" one (status is only visible partially) If "workingctx" object is shared between some procedures indirectly referring "ctx._status", this incorrect caching may cause unexpected result: for example, "ctx._status" is used via "manifest()", "files()" and so on. To cache "self._status" correctly at "workingctx.status", this patch overwrites "self._status" in "workingctx._buildstatus" only when: - status is calculated against the parent of the working directory, and - specified "match" is "always" one This patch can be applied (and effective) only on default branch, because procedure around "basectx.status" is much different between stable and default: for example, overwriting "self._status" itself is executed not in "workingctx._buildstatus" but in "workingctx._poststatus", on stable branch.

filectx.parents: also fetch the filelog of rename source too we are going to need this filelog for the linkrev adjustment, so we better normalise the list and have the filelog in all case. This is done in a previous changeset to help readability.

Added signature for changeset 1265a3a71d75

Added tag 3.2.4 for changeset 1265a3a71d75

largefiles: backout f72d73937853 - linear updates handle m -> a differently f72d73937853 introduced a significant performance regression: All largefiles were marked 'normallookup' by linear (or noop) updates and had to be rehashed by the next command. The previous change introduced a different solution to the problem f72d73937853 solved and we can thus back it out again.

largefiles: mark lfile as added in lfdirstate when the standin is added This is an alternative solution to the problem addressed by f72d73937853. This implementation has the advantage that it doesn't mark clean largefiles as normallookup. We can thus avoid repeated rehashing of all largefiles when f72d73937853 is backed out. This implementation use the existing 'lfmr' actions that 23fe278bde43 introduced for handling another part of the same cases.

tests: add test coverage for lfdirstate invalidation of linear update f72d73937853 introduced a significant performance regression: All largefiles are marked 'normallookup' in lfdirstate by linear (or noop) updates and has to be rehashed by the next command. To avoid such regressions, keep an eye on the dirstate content after a plain 'hg up'.

test-subrepo-git: ignore global git config This was causing a test failure for people with company-wide settings. Still need a way to ignore local config.

tests/autodiff.py: explicitly only honor feature diffopts This test extension manages the opts it cares about on its own anyway.

cmdutil.changeset_printer: explicitly honor all diffopts This is used in hg log -p so the output is expected to be the same as that of hg diff.

export: explicitly honor all diffopts This is slightly more controversial than diff, but we hope that HGPLAIN=1 covers all the format-breaking ones. A possible alternative here that breaks BC is to honor all opts except the whitespace ones.

webcommands.annotate: explicitly only honor whitespace diffopts The whitespace ones are the only ones the annotate logic cares about anyway, so there's no visible impact.

filectx.parents: filter nullrev parent sooner We are going to introduce a linkrev-correction phases when computing parents. It will be more convenient to have the nullid parent filtered out earlier. I had to make a minimal adjustment to the rename handling logic to keep it functional. That logic have been documented in the process since it took me some time to check all the cases out.

context: catch FilteredRepoLookupError instead of RepoLookupError Now that we have a more specialised exception, lets use it when we meant to catch the more specialised case.

narrowmatcher: propagate the rel() method The full path is propagated to the original match object since this is often used directly for printing a file name to the user. This is cleaner than requiring each caller to join the prefix with the file name prior to calling it, and will lead to not having to pass the prefix around separately. It is also consistent with the bad() and abs() methods in terms of the required input. The uipath() method now inherits this path building property. There is no visible change in path style for rel() because it ultimately calls util.pathto(), which returns an os.sep based path. (The previous os.path.join() was violating the documented usage of util.pathto(), that its third parameter be '/' separated.) The doctest needed to be normalized to '/' separators to avoid test differences on Windows, now that a full path is returned for a short filename. The test changes are to drop globs that are no longer necessary when printing an absolute file in a subrepo, as returned from match.uipath(). Previously when os.path.join() was used to add the prefix, the absolute path to a file in a subrepo was printed with a mix of '/' and '\'. The absolute path for a file not in a subrepo, as returned from match.uipath(), is still purely '/' based.

match: add the abs() method This is a utility to make it easier for subrepos to convert a file name to the full path rooted at the top repository. It can replace the various path joining lambdas, and doesn't require the prefix to be passed into the method that wishes to build such a path. The name is derived from the following pattern in annotate() and other places: name = ((pats and rel) or abs) The pathname separator is not os.sep in part to avoid confusion with variables named 'abs' or similar that _are_ '/' separated, and in part because some methods like cmdutils.forget() and maybe cmdutils.add() need to build a '/' separated path to the root repo. This can replace the manual path building there.

merge with stable

posix: quote the specified string only when it may have to be quoted This patch makes "posix.shellquote" examine the specified string and quote it only when it may have to be quoted for safety, like as the previous patch for "windows.shellquote". In fact, on POSIX environment, quoting itself doesn't cause issues like issue4463. But (almost) equivalent quoting policy can avoid examining test result differently on POSIX and Windows (even though showing command line with "%r" causes such examination in "test-extdiff.t"). The last hunk for "test-extdiff.t" in this patch isn't needed for the previous patch for "windows.shellquote", because the code path of it is executed only "#if execbit" (= avoided on Windows).

windows: quote the specified string only when it has to be quoted Before this patch, "windows.shellquote" (as used as "util.shellquote") always quotes specified strings with double quotation marks, for external process invocation. But some problematic applications can't work correctly, when command line arguments are quoted: see issue4463 for detail. On the other hand, quoting itself is needed to specify arguments containing whitespaces and/or some special characters exactly. This patch makes "windows.shellquote" examine the specified string and quote it only when it may have to be quoted for safety.

extdiff: rename the name of an argument for readability To reduce amount of changes for review-ability, previous patch uses "args" as argument name of "dodiff()", even though "args" includes also the name of command to be executed (or full-path of it). This patch replaces "args" by more appropriate name "cmdline".

extdiff: avoid unexpected quoting arguments for external tools (issue4463) Before this patch, all command line arguments for external tools are quoted by the combination of "shlex.split" and "util.shellquote". But this causes some problems. - some problematic commands can't work correctly with quoted arguments For example, 'WinMerge /r ....' is OK, but 'WinMerge "/r" ....' is NG. See also below for detail about this problem. https://bitbucket.org/tortoisehg/thg/issue/3978/ - quoting itself may change semantics of arguments For example, when the environment variable CONCAT="foo bar baz': - mydiff $CONCAT => mydiff foo bar baz (taking 3 arguments) - mydiff "$CONCAT" => mydiff "foo bar baz" (taking only 1 argument) For another example, single quoting (= "util.shellquote") on POSIX environment prevents shells from expanding environment variables, tilde, and so on: - mydiff "$HOME" => mydiff /home/foobar - mydiff '$HOME' => mydiff $HOME - "shlex.split" can't handle some special characters correctly It just splits specified command line by whitespaces. For example, "echo foo;echo bar" is split into ["echo", "foo;echo", "bar"]. On the other hand, if quoting itself is omitted, users can't specify options including space characters with "--option" at runtime. The root cause of this issue is that "shlex.split + util.shellquote" combination loses whether users really want to quote each command line elements or not, even though these can be quoted arbitrarily in configurations. To resolve this problem, this patch does: - prevent configurations from being processed by "shlex.split" and "util.shellquote" only (possibly) "findexe"-ed or "findexternaltool"-ed command path is "util.shellquote", because it may contain whitespaces. - quote options specified by "--option" via command line at runtime This patch also makes "dodiff()" take only one "args" argument instead of "diffcmd" and "diffopts. It also omits applying "util.shellquote" on "args", because "args" should be already stringified in "extdiff()" and "mydiff()". The last hunk for "test-extdiff.t" replaces two whitespaces by single whitespace, because change of "' '.join()" logic causes omitting redundant whitespaces.