largefiles: enable subrepo support for remove Previously, remove failed when operating on a largefile in a subrepo, stating that the file is untracked.

transplant: properly skip empty changeset (issue4423) If resolving a merge conflict result in an empty changesets, we now properly skip the changeset instead of crashing. Original patch from Robert Collins <robertc@robertcollins.net>.

color: add support for colorizing git subrepo diffs

tests: fix test-casefolding.t output This change started with a387b0390082.

synthrepo: new filenames must not also be new directories, and vice-versa When generating many new files into a set of many possible new directories, there is the possibility that the same path is chosen as both file and directory. How likely this is depends on the size of the dictionary used, the generated directory structure and the number of generated files.

status: don't override _buildstatus() in workingcommitctx Now that the caching into _status is done in workingctx._dirstatestatus(), which workingcommitctx._dirstatestatus() does not call, there is no caching to prevent in _buildstatus(), so stop overriding it.

status: cache dirstate status in _dirstatestatus() Since it's only the dirstate status we cache, it makes more sense to cache it in the _dirstatestatus() method. Note that this change means the dirstate status will also be cached when status is requested between the working copy and some other revision, while we currently only cache the result if exactly the status between the working copy and its parent is requested.

localrepo: add ignoremissing parameter to branchtip Previously, in the namespaces api, the only caller of branchtip was singlenode which happened to raise the same exception that branchtip raised: KeyError. This is a minor change but will allow upcoming patches to use repo.branchtip to not raise an exception if a branch doesn't exist. After that, it will be possible for extensions to use the namespace api in a stable way.

namespaces: add method to return a list of nodes for a given name This is a helpful method that some extensions can make use of (e.g. for custom revsets); currently not used in core.

namespaces: add test for log Now that we have enough features in the namespaces api, we add a test for it.

log: use new namespaces api to display names The only caveat here is that branches must be displayed first due to backwards compatibility. The order of namespaces is defined to be the 'update' order which, unfortunately, is not the same as log output order. It's worth mentioning that the log output is still translated the same as before since we are formating our strings the same way: # i18n: column positioning for "hg log" _("bookmark: %s\n") % bookmark becomes tname = _(("%s:" % ns.templatename).ljust(13) + "%s\n") % name when name == 'bookmark'. The ljust(13) keeps the strings and whitespace equal. Adding a new namespace is even easier now because the log output code doesn't need to change. A future programmer would just need to add the string to the corresponding .po file (which is the same as they would have had to do previously).

tests: add a i18n translation test for log output Upcoming patches will change the way that log output is generated so we add a test to ensure that the words 'branches', 'bookmarks', and 'tags' are still translated.

filectx: fix annotate to not directly instantiate filectx 28a302e9225d changed basefilectx.annotate() to directly instantiate new filectx's instead of going through self.filectx(), this breaks extensions that replace the filectx class, and would also break future uses that would need memfilectx's.

largefiles: don't print files as both large and normal in addremove dryruns

largefiles: return the list of added files from addlargefiles() This will be used to exclude largefile candidates from the normal file matcher, which will allow add and addremove dryruns to not print a file as both a normal and a large file.

largefiles: align the output messages for an added file with core methods Core addremove prints the file relative to cwd only if patterns are provided to the command. Core add always prints relative to cwd. Also, both methods print the subrepo prefix when needed. The 'already a largefile' doesn't have an analog in core, but follows the same rules for consistency.

largefiles: align the output messages for a removed file with core methods Both cmdutil.remove() and scmutil.addremove() require verbose mode or an inexact match to print the filename. Core addremove also prints the file relative to cwd only if patterns are provided to the command. And finally, both methods print the subrepo prefix when needed.

revset: use '%' as an operator for 'only' With this patch, we can make it much easier to specify 'only(A,B)' -> A%B. Similarly, 'only(A)' -> A%. On Windows, '%' is a semi-reserved symbol in the following way: using non-bash shells (e.g. cmd.exe but NOT PowerShell, ConEmu, and cmder), %var% is only expanded when 'var' exists and is surrounded by '%'. That only leaves batch scripts which could prove to be problematic. I posit that this isn't a big issue because any developer of batch scripts already knows that to use '%' one needs to escape it by using a double '%%'. Alternatives to '%' could be '=' but that might be limiting our future if we ever decide to use temporary assignments in a revset.

transaction: support for callbacks during abort Previous transaction work added callbacks to be called during regular transaction commit/close. As part of refactoring Mozilla's pushlog extension (an extension that opens a SQLite database and tries to tie its transaction semantics to Mercurial's transaction), I discovered that the new transaction APIs were insufficient to avoid monkeypatching transaction instance internals. Adding a callback that is called during transaction abort removes the necessity for monkeypatching and completes the API.

debugnamecomplete: use new name api Instead of hardcoding a list of places to check, we use the new repo.names api to get a list of potential names to complete.

debugnamecomplete: rename from debuglabelcomplete Now that we have decided on the use of 'name' instead of 'label' we rename this function accordingly. The old method 'debuglabelcomplete' has been left as a deprecated command so that current scripts don't break.

namespaces: add __iter__ and iteritems methods Iterating over all the namespaces is a common operation, naturally, so we add those methods now. Since we are using a sorted dictionary, this method just calls the underlying __iter__ or iteritems method.

namespaces: add 'listnames' property Currently, we have no way to list all the names in a given namespace. This is needed for things such as tab completion. Future patches will use this patch for exactly that purpose.

test-bundle2-exchange: create temp script in $TESTTMP, not $TESTDIR The bundle2-pushkey-hook.sh script is currently created in $TESTTMP, and leaves an untracked file in that directory (tests/) after running.

manifest: drop withflags() method, which is now unused

context: use new manifest.diff(clean=True) support This further simplifies the status code. This simplification comes at a slight performance cost for `hg export`. Before, on mozilla-central: perfmanifest tip ! wall 0.265977 comb 0.260000 user 0.240000 sys 0.020000 (best of 38) perftags ! result: 162 ! wall 0.007172 comb 0.010000 user 0.000000 sys 0.010000 (best of 403) perfstatus ! wall 0.422302 comb 0.420000 user 0.260000 sys 0.160000 (best of 24) hgperf export tip ! wall 0.148706 comb 0.150000 user 0.150000 sys 0.000000 (best of 65) after, same repo: perfmanifest tip ! wall 0.267143 comb 0.270000 user 0.250000 sys 0.020000 (best of 37) perftags ! result: 162 ! wall 0.006943 comb 0.010000 user 0.000000 sys 0.010000 (best of 397) perfstatus ! wall 0.411198 comb 0.410000 user 0.260000 sys 0.150000 (best of 24) hgperf export tip ! wall 0.173229 comb 0.170000 user 0.170000 sys 0.000000 (best of 55) The next set of patches introduces a new manifest type implemented almost entirely in C, and more than makes up for the performance hit incurred in this change.

manifest: add optional recording of clean entries to diff This makes manifest slightly easier to use for status code.

context: use manifest.diff() to compute most of status We can do a little tiny bit better by enhancing manifest.diff to optionally include files that are in both sides. This will be done in a followup patch.

trydiff: replace dodiff=True/'binary' by binarydiff=False/True

trydiff: replace 'dodiff = False' by 'continue' The 'dodiff' variable is initialized to True and may later be set to either False or "binary". When it's set to False, we skip everything after that point, so we can simplify by instead continue-ing (the loop). We can then also drop the 'if dodiff', since it will always be true.

trydiff: make addindexmeta() unconditionally add index meta The conditional-ness is not clear from the name and there is only one caller, so it's clearer to check on the call site. Moving it also makes addindexmeta() no longer close on the 'opts' variable.

trydiff: remove unused code for git index of "combined diff" We only ever produce diffs with one pre- and one post-image, so remove the code for displaying "index" lines for combined diffs.

diff: remove dead assignment

localrepo: show headline notes in commitctx before showing filenames commitctx already showed notes with filenames but didn't provide any context. It is just as relevant to know when manifest or changelog is committed. So, in addition to filenames, also show headlines 'committing files:', 'committing manifest' and 'committing changelog'.

bundle: when verbose, show what takes up the space in the generated bundle This is kind of similar to the debugbundle command but gives summarized actual uncompressed number of bytes when creating the bundle. The numbers are as usable as the bundle format is efficient. Hopefully bundle2 will make it a better indicator of actual entropy. This is useful when accepting pull requests to assess whether the repo size increase seems reasonable for the diff before pushing stuff upstream, It has helped me catching large files that should have been committed as largefiles but was committed as regular files in intermediate changesets. This output doesn't combine well with debug output so we only enable it when verbose without debug.

setdiscovery: factorize similar sampling code We are using full sampling of 'fullsamplesize' in both case. The only difference is the debug message. So we factorise the sampling code and put the message in an extra conditional. This is going to help making changes around the sampling logic. Such changes are needed to improve discovery performance on highly headed repository.

setdiscovery: drop shadowed 'undecided' assignment The 'undecided' variable was never used before being overwritten a few lines later.

hgweb: extract changelist entry generation into own function I want to supplement changelist entries (used by shortlog and changelog endpoints) with custom metadata from an extension. i.e. I have extra per-changeset metadata that I wish to make available to templates so it can be rendered on hgweb. To facilitate this, I've extracted the logic for creating a changeset data structure into its own function, where it can be wrapped by extensions. Ideally, hgweb would use the same templater as the command line and have full access to templatekw.keywords. But that's a lot of work. This patch gets us some of the benefit without all the work. Many other hgweb commands could benefit from similar refactorings. I'm going to wait to see how this patch is received before I go crazy on extracting inline functions.

templates: use CSS classes for diff styling Use of inline style for diff styling led to significant browser memory usage on large diffs. Moving the styling into CSS classes corrects this. This patch is based on work from https://bugzilla.mozilla.org/show_bug.cgi?id=766952 and https://hg.mozilla.org/hgcustom/version-control-tools/rev/2c355a580af6

templates: remove unnecessary <span> The <span> on diffline was useless. It was only making browsers work harder. Remove it.

revset: introduce new operator "##" to concatenate strings/symbols at runtime Before this patch, there is no way to concatenate strings at runtime. For example, to search for the issue ID "1234" in descriptions against all of "issue 1234", "issue:1234", issue1234" and "bug(1234)" patterns, the revset below should be written fully from scratch for each issue ID. grep(r"\bissue[ :]?1234\b|\bbug\(1234\)") This patch introduces new infix operator "##" to concatenate strings/symbols at runtime. Operator symbol "##" comes from the same one of C pre-processor. This concatenation allows parametrizing a part of strings in revset queries. In the case of example above, the definition of the revset alias using operator "##" below can search issue ID "1234" in complicated patterns by "issue(1234)" simply: issue($1) = grep(r"\bissue[ :]?" ## $1 ## r"\b|\bbug\(" ## $1 ## r"\)") "##" operator does: - concatenate not only strings but also symbols into the string Exact distinction between strings and symbols seems not to be convenience, because it is tiresome for users (and "revset.getstring" treats both similarly) For example of revset alias "issue()", "issue(1234)" is easier than "issue('1234')". - have higher priority than any other prefix, infix and postfix operators (like as "##" of C pre-processor) This patch (re-)assigns the priority 20 to "##", and 21 to "(", because priority 19 is already assigned to "-" as prefix "negate".

largefiles: pass a matcher instead of a raw file list to removelargefiles() This is consistent with addlargefiles(), and will make it easier to get the paths that are printed correct when recursing into subrepos or invoking from outside the repository. It also now restricts the path that the addremove is performed on if a path is given, as is done with normal files. The repo.status() call needs to exclude clean files when performing an addremove, because the addremove override method calling this used to pass the list of files to delete, which caused the matcher to only consider those files in building the status list. Now the matcher is restricted only to the extent that the caller requested- usually directories if at all. There's no reason for addremove to care about clean files anyway- we don't want them deleted.

hgweb: allow viewing diffs against p1 or p2 for merge commits (issue3904) This adds UI portion of the feature that has resided in mercurial since 2012. Back then the interface was added together with the code, but was shortly backed out because it was deemed "not ready". Code, however, stayed. For the original feature and its implementation, see issue2810 and d605a82cf189. In short, the backed-out interface had two outstanding issues: 1. it was introducing an entirely new term (baseline) and 2. it was present on every changeset's page, even for changesets with 1 parent (or no parents), which didn't make sense This patch implements a hopefully better interface because: 1. it uses the usual terms (diff) and 2. it only shows up when there actually are 2 parents.

namespaces: remove templatename method on the namespaces object Now that there is an object for each individual namespace, we use the templatename property on that object instead of the method on the collection of namespaces.

namespaces: remove names method on the namespaces object Now that there is an object for each individual namespace, we use that instead of the method on the collection of namespaces.

templatekw: update namespace calls Previous patches changed the namespace api to be more of an object-oriented approach. This patch updates the template function to use said api changes.

namespaces: add __getitem__ property Since the namespaces object uses an underlying (sorted) dictionary to store the namespaces, it makes sense to expose this to naturally gain access to those namespaces.

merge with stable

cmdutil.jsonchangeset: properly compute added and removed files jsonchangeset._show() was computing the reverse status of the current changeset. As a result, added files were showing up as removed and removed files were showing up as adds. There were existing tests for this code and they were flat out wrong.

largefiles: convert addlargefiles() to vfs

rebase: clarify comment about merge ancestor when rebasing merges The code for picking a merge ancestor when rebasing merges had a long and incorrect comment. The comment would perhaps have been fine as commit message but does not make the code more readable or maintainable and is a bad substitute for correct and readable code. The correct essense of the comment is quite trivial: a merge of an ancestor of the rebase destination and an 'outside' revision can be rebased as if it was a linear change, using 'destination ancestor parent' as base and pretty much ignoring the 'outside' revision. The code path where the comment is placed is however also used for other kinds of merge rebases. The comment is thus not really correct and not helpful. I think it would be better to drop the comment and rewrite the code.

status: don't list files as both clean and deleted Tracked files that are deleted should always be reported as such, no matter what their state was in earlier revisions. This is encoded in in two conditions in the loop in basectx._buildstatus() for modified and added files, but the check is missing for clean files. We should check for clean files too, but instead of adding the check in a third place, move it earlier and skip most of the loop body for deleted files.

status: don't list files as both removed and deleted When calculating status involving the working copy and a revision other than the parent of the working copy, the files that are not in the working context manifest ('mf2' in the basectx._buildstatus()) will be reported as removed (note that deleted files _are_ in the working context manifest). However, if the file is reported as deleted in the dirstate, it will get that status too (as shown by failing tests). Fix by removing deleted files from the 'removed' list after the main loop in _buildstatus().

revset-filelog: handle hidden linkrev for file missing for head (issue4490) The fix for linkrev pointing to hidden revision was crashing when the file was missing from head's manifest. We now properly handle this case. (yes I feel silly)

run-tests: automatically add (glob) to "saved backup bundle to" lines Avoid spending too much time adding (glob) after running run-tests -i. This doesn't handle all cases but it helps. The run-tests tests add a bit of escaping of trailing (glob) in the output to avoid interference from the outer test runner. The regexp for matching the output lines contains a group for making multiline substitute in a way that works with Python before 2.7.

transaction: use the right location when cleaning up backup file (issue4479) The location variable fetch from the loop and the one used to actually fetch it mismatched. We fix the name to ensure file outside of store are cleaned up.

merge with stable

revset: delay showing parse error for the revset alias until it is referred Before this patch, a problematic revset alias aborts execution immediately, even if it isn't referred in the specified revset. If old "hg" may be used too (for example, bisecting Mercurial itself), it is also difficult to write alias definitions using features newly introduced by newer "hg" into configuration files, because such alias definitions cause unexpected abortion at parsing revset aliases with old "hg". This patch delays showing parse error for the revset alias until it is actually referred at runtime. This patch detects referring problematic aliases in "_expandaliases" by examination of "revsetalias.error", which is initialized with the error message only when parsing fails. For usability, this patch also warns about problematic aliases, even if they aren't referred at runtime. This should help users to know potential problems in their alias definitions earlier.

revset: drop pre-lazyset optimization for stringset of subset == entire repo It was introduced at e44ebd2a142a, where spanset.__contains__() did not exist. Nowadays, we have to pay huge penalty for len(subset). The following example showed that OR operation could be O(n * m^2) (n: len(repo), m: number of OR operators, m >= 2) probably because of filteredset.__len__. revset #0: 0|1|2|3|4|5|6|7|8|9 0) wall 8.092713 comb 8.090000 user 8.090000 sys 0.000000 (best of 3) 1) wall 0.445354 comb 0.450000 user 0.430000 sys 0.020000 (best of 22) 2) wall 0.000389 comb 0.000000 user 0.000000 sys 0.000000 (best of 7347) (0: 3.2.4, 1: 3.1.2, 2: this patch)

keyword: update copyright year

keyword: use vfs.reljoin and util.unlinkpath to remove kwdemo

largefiles: properly sync lfdirstate after removing largefiles The more aggressive synchronization of lfdirstate that was backed out in 1265a3a71d75 masked the problem where lfdirstate would hold an 'R' for a largefile that was added and then removed without a commit between. We could just conditionally call lfdirstate.drop() or lfdirstate.remove() here, but this also properly updates lfdirstate if the standin doesn't exist for the file somehow (i.e. call drop instead of remove). Without this change, the precommit status in the commit command immediately after the test change lists the removed (and never committed) largefile as 'R'. It can also lead to situations where the status command reports the same, long after the commit [1]. [1] http://www.selenic.com/pipermail/mercurial-devel/2015-January/065153.html

linkrev-filelog: handle filtered linkrev with no visible children (issue4307) If the file revision with a filtered linkrev does not have any (unfiltered) children, we cannot use it to bound the search for another introduction. Instead, we have to look at the file revision used by each head changeset. If one of them uses this file revision, we know there is another occurrence and we have a starting point. See inline comments for details. Adding some kind of permanent reference of all the introductions of a file revision instead of just the first one would be much better. But this is more difficult. I hope to take that into account in the next repository format.

linkrev: work around linkrev to filtered entry in 'filelog' revset This revset is used by 'hg log FILENAME'. This prevent bugs when used on a repository with hidden revisions. Instead of just discarding file revisions whose linkrevs point to filtered revisions, we put them aside and post-process them trying to find a non-filtered introduction. See inline documentation for details about how it works. This only fixes some of the problems. Once again, more will be needed when we can cannot rely on child revisions of a file to find linkrev-shadowned revisions. A test is added for 'hg log' catching such cases.

namespaces: update documentation and code indentation The previous patch changed just the functionality, while this patch focuses on the documentation and indentation to keep review simple.

namespaces: use namespace object instead of dictionary This isn't as bad as the diff seems, it only looks like scary. In this patch, we use the 'namespace' object instead of accessing keys in a dictionary. This required the 'templatename' and 'names' method to change their implementation. Later, we will remove these functions entirely due to a better api.

namespaces: copy implementation to new namespace object

namespaces: add a namespace object Currently, we use a dictionary object to store the namespace properties. This is python so let's use an object. This will allow us to be more flexible in our method signatures in the future.

localrepo: use vfs.reljoin rather than os.path.join in the localrepository class The removes the last remaining usage of os.path.join inside teh localrepository class.

largefiles: drop a redundant check for patterns when removing without --after This is in the way of passing a matcher to removelargefiles(). This method is called in exactly two places- first in overrides.addremove() (but only if the pattern list passed to it is not empty), and second in the commands.remove() override. But since the latter calls commands.remove() first, which also does this check, it isn't needed here.

context: override _dirstatestatus in workingcommitctx for correct matching Before this patch, the result of "status()" on "workingcommitctx" may incorrectly contain files other than ones to be committed, because "workingctx._dirstatestatus()" returns the result of "dirstate.status()" directly. For correct matching, this patch overrides "_dirstatestatus" in "workingcommitctx" and makes it return matched files only in "self._status". This patch uses empty list for "deleted", "unknown" and "ignored" of status, because status between "changectx"s also makes them empty.

context: avoid breaking already fixed self._status at ctx.status() Before this patch, "status()" on "workingcommitctx" with "always match" object causes breaking "self._status" in "workingctx._buildstatus()", because "workingctx._buildstatus()" caches the result of "dirstate.status()" into "self._status" for efficiency, even though it should be fixed at construction for committing. For example, template function "diff()" without any patterns in "committemplate" implies "status()" on "workingcommitctx" with "always match" object, via "basectx.diff()" and "patch.diff()". Then, broken "self._status" causes committing unexpected files. To avoid breaking already fixed "self._status" at "ctx.status()", this patch overrides "_buildstatus" in "workingcommitctx". This patch doesn't write out the result of template function "diff()" in "committemplate" in "test-commit.t", because matching against files to be committed still has an issue fixed in subsequent patch.

context: add workingcommitctx for exact context to be committed Before this patch, "workingctx" is also used for the context to be committed. But "workingctx" works incorrectly in some cases. For example, even when only some of changed files in the working directory are committed, "status()" on "workingctx" object for committing recognizes files not to be committed as changed, too. As the preparation for fixing these issues, this patch chooses adding new class "workingcommitctx" for exact context to be committed, because switching by the flag (like "self._fixedstatus" or so) in some code paths of "workingctx" is less readable and maintenancable.

context: make unknown/ignored/clean of cached status empty for equivalence Before this patch, "workingctx.status" caches the result of "dirstate.status" directly into "self._status". But "dirstate.status" is invoked with False "list*" arguments in normal "self._status" accessing route, and this makes "unknown"/"ignored"/"clean" of status empty. This may cause unexpected result of code paths internally accessing to them (accessors for external usage are already removed by previous patch). This patch makes "unknown"/"ignored"/"clean" of cached status empty for equivalence. Making them empty is executed only when at least one of "unknown", "ignored" or "clean" has files, for efficiency.

templatefilters.json: stabilize output The json filter was previously iterating over keys in an object in an undefined order. Let's throw a sorted() in there so output is consistent. It's somewhat frightening that there are no tests for the json filter. Subsequent commits will add them, so we pass on the opportunity to add them here.

templatefilters.json: call functions The "changeset" template from hgweb is using a lambda in the "diffsummary" key. In preparation for enabling JSON output from hgweb, teach the json filter how to call functions.

merge with stable

linkrev: also adjust linkrev when bootstrapping annotate (issue4305) The annotate logic now use the new 'introrev' method to bootstrap its traversal. This catches issues from linkrev-shadowing of the changeset introducing the version of a file in source changeset. More tests have been added to display pathological cases.

linkrev: also adjust linkrev when bootstrapping 'follow' revset The follow revset (used by `hg log --follow`) now uses the new 'introrev' method to bootstrap its traversal. This catches issues from linkrev-shadowing of the changesets introducing the version of a file in source changeset. A new test has been added to display pathological cases. Another test is affected because it was meant to test this behavior but actually failed to do so for half of the output. The output are now similar.

linkrev: introduce an 'introrev' method on filectx The previous changeset properly fixed the ancestors computation, but we need to ensure that the initial filectx is also using the right changeset. When asking for log or annotation from a certain point, the first step is to define the changeset that introduced the current file version. We cannot just pick the "starting point" changesets as it may just "use" the file revision, unchanged. Currently, we were using 'linkrev' for this purpose, but this exposes us to unexpected branch-jumping when the revision introducing the starting point version is itself linkrev-shadowed. So we need to take the topology into account again. Therefore, we introduce an 'introrev' function, returning the changeset which introduced the file change in the current changeset. This function will be used to fix linkrev-related issues when bootstrapping 'hg log --follow' and 'hg annotate'. It reuses the '_adjustlinkrev' function, extending it to allow introspection of the initial changeset too. In the previous usage of the '_adjustlinkrev' the starting rev was always using a children file revisions, so it could be safely ignored in the search. In this case, the starting point is using the revision of the file we are looking, and may be the changeset we are looking for.

filectx.parents: enforce changeid of parent to be in own changectx ancestors Because of the way filenodes are computed, you can have multiple changesets "introducing" the same file revision. For example, in the changeset graph below, changeset 2 and 3 both change a file -to- and -from- the same content. o 3: content = new | | o 2: content = new |/ o 1: content = old In such cases, the file revision is create once, when 2 is added, and just reused for 3. So the file change in '3' (from "old" to "new)" has no linkrev pointing to it). We'll call this situation "linkrev-shadowing". As the linkrev is used for optimization purposes when walking a file history, the linkrev-shadowing results in an unexpected jump to another branch during such a walk.. This leads to multiple bugs with log, annotate and rename detection. One element to fix such bugs is to ensure that walking the file history sticks on the same topology as the changeset's history. For this purpose, we extend the logic in 'basefilectx.parents' so that it always defines the proper changeset to associate the parent file revision with. This "proper" changeset has to be an ancestor of the changeset associated with the child file revision. This logic is performed in the '_adjustlinkrev' function. This function is given the starting changeset and all the information regarding the parent file revision. If the linkrev for the file revision is an ancestor of the starting changeset, the linkrev is valid and will be used. If it is not, we detected a topological jump caused by linkrev shadowing, we are going to walk the ancestors of the starting changeset until we find one setting the file to the revision we are trying to create. The performance impact appears acceptable: - We are walking the changelog once for each filelog traversal (as there should be no overlap between searches), - changelog traversal itself is fairly cheap, compared to what is likely going to be perform on the result on the filelog traversal, - We only touch the manifest for ancestors touching the file, And such changesets are likely to be the one introducing the file. (except in pathological cases involving merge), - We use manifest diff instead of full manifest unpacking to check manifest content, so it does not involve applying multiple diffs in most case. - linkrev shadowing is not the common case. Tests for fixed issues in log, annotate and rename detection have been added. But this changeset does not solve all problems. It fixes -ancestry- computation, but if the linkrev-shadowed changesets is the starting one, we'll still get things wrong. We'll have to fix the bootstrapping of such operations in a later changeset. Also, the usage of `hg log FILE` without --follow still has issues with linkrev pointing to hidden changesets, because it relies on the `filelog` revset which implement its own traversal logic that is still to be fixed. Thanks goes to: - Matt Mackall: for nudging me in the right direction - Julien Cristau and Rémi Cardona: for keep telling me linkrev bug were an evolution show stopper for 3 years. - Durham Goode: for finding a new linkrev issue every few weeks - Mads Kiilerich: for that last rename bug who raise this topic over my anoyance limit.

context: remove unreliable accessor methods from committablectx There are two caching routes for (propertycache-ed) "_status" below in committablectx: - invoking "status()": "dirstate.status()" is invoked, and the result of it is cached into "_status". In this case, any of "listignored", "listclean" and "listunknown" may be True. - accessing "_status" directly before "status()": Own "status()" is invoked, but all of "listignored", "listclean" and "listunknown" arguments are False, in this case. "ignored"/"clean"/"unknown" accessor methods of "committablectx" use corresponded fields of "_status", but these fields aren't reliable, because these fields are empty when: - "_status" method is executed before accessors, or - "status()" is executed with "list*=False" before accessors In addition to it, these accessors aren't used in the recent Mercurial implementation. At least, removing them doesn't cause any test failures.

context: cache self._status correctly at workingctx.status Before this patch, "workingctx.status" always replaces "self._status" by the recent result, even though: - status isn't calculated against the parent of the working directory, or - specified "match" isn't "always" one (status is only visible partially) If "workingctx" object is shared between some procedures indirectly referring "ctx._status", this incorrect caching may cause unexpected result: for example, "ctx._status" is used via "manifest()", "files()" and so on. To cache "self._status" correctly at "workingctx.status", this patch overwrites "self._status" in "workingctx._buildstatus" only when: - status is calculated against the parent of the working directory, and - specified "match" is "always" one This patch can be applied (and effective) only on default branch, because procedure around "basectx.status" is much different between stable and default: for example, overwriting "self._status" itself is executed not in "workingctx._buildstatus" but in "workingctx._poststatus", on stable branch.

filectx.parents: also fetch the filelog of rename source too we are going to need this filelog for the linkrev adjustment, so we better normalise the list and have the filelog in all case. This is done in a previous changeset to help readability.

Added signature for changeset 1265a3a71d75

Added tag 3.2.4 for changeset 1265a3a71d75

largefiles: backout f72d73937853 - linear updates handle m -> a differently f72d73937853 introduced a significant performance regression: All largefiles were marked 'normallookup' by linear (or noop) updates and had to be rehashed by the next command. The previous change introduced a different solution to the problem f72d73937853 solved and we can thus back it out again.

largefiles: mark lfile as added in lfdirstate when the standin is added This is an alternative solution to the problem addressed by f72d73937853. This implementation has the advantage that it doesn't mark clean largefiles as normallookup. We can thus avoid repeated rehashing of all largefiles when f72d73937853 is backed out. This implementation use the existing 'lfmr' actions that 23fe278bde43 introduced for handling another part of the same cases.

tests: add test coverage for lfdirstate invalidation of linear update f72d73937853 introduced a significant performance regression: All largefiles are marked 'normallookup' in lfdirstate by linear (or noop) updates and has to be rehashed by the next command. To avoid such regressions, keep an eye on the dirstate content after a plain 'hg up'.

test-subrepo-git: ignore global git config This was causing a test failure for people with company-wide settings. Still need a way to ignore local config.

tests/autodiff.py: explicitly only honor feature diffopts This test extension manages the opts it cares about on its own anyway.

cmdutil.changeset_printer: explicitly honor all diffopts This is used in hg log -p so the output is expected to be the same as that of hg diff.

export: explicitly honor all diffopts This is slightly more controversial than diff, but we hope that HGPLAIN=1 covers all the format-breaking ones. A possible alternative here that breaks BC is to honor all opts except the whitespace ones.

webcommands.annotate: explicitly only honor whitespace diffopts The whitespace ones are the only ones the annotate logic cares about anyway, so there's no visible impact.

filectx.parents: filter nullrev parent sooner We are going to introduce a linkrev-correction phases when computing parents. It will be more convenient to have the nullid parent filtered out earlier. I had to make a minimal adjustment to the rename handling logic to keep it functional. That logic have been documented in the process since it took me some time to check all the cases out.

context: catch FilteredRepoLookupError instead of RepoLookupError Now that we have a more specialised exception, lets use it when we meant to catch the more specialised case.

narrowmatcher: propagate the rel() method The full path is propagated to the original match object since this is often used directly for printing a file name to the user. This is cleaner than requiring each caller to join the prefix with the file name prior to calling it, and will lead to not having to pass the prefix around separately. It is also consistent with the bad() and abs() methods in terms of the required input. The uipath() method now inherits this path building property. There is no visible change in path style for rel() because it ultimately calls util.pathto(), which returns an os.sep based path. (The previous os.path.join() was violating the documented usage of util.pathto(), that its third parameter be '/' separated.) The doctest needed to be normalized to '/' separators to avoid test differences on Windows, now that a full path is returned for a short filename. The test changes are to drop globs that are no longer necessary when printing an absolute file in a subrepo, as returned from match.uipath(). Previously when os.path.join() was used to add the prefix, the absolute path to a file in a subrepo was printed with a mix of '/' and '\'. The absolute path for a file not in a subrepo, as returned from match.uipath(), is still purely '/' based.

match: add the abs() method This is a utility to make it easier for subrepos to convert a file name to the full path rooted at the top repository. It can replace the various path joining lambdas, and doesn't require the prefix to be passed into the method that wishes to build such a path. The name is derived from the following pattern in annotate() and other places: name = ((pats and rel) or abs) The pathname separator is not os.sep in part to avoid confusion with variables named 'abs' or similar that _are_ '/' separated, and in part because some methods like cmdutils.forget() and maybe cmdutils.add() need to build a '/' separated path to the root repo. This can replace the manual path building there.

merge with stable

posix: quote the specified string only when it may have to be quoted This patch makes "posix.shellquote" examine the specified string and quote it only when it may have to be quoted for safety, like as the previous patch for "windows.shellquote". In fact, on POSIX environment, quoting itself doesn't cause issues like issue4463. But (almost) equivalent quoting policy can avoid examining test result differently on POSIX and Windows (even though showing command line with "%r" causes such examination in "test-extdiff.t"). The last hunk for "test-extdiff.t" in this patch isn't needed for the previous patch for "windows.shellquote", because the code path of it is executed only "#if execbit" (= avoided on Windows).

windows: quote the specified string only when it has to be quoted Before this patch, "windows.shellquote" (as used as "util.shellquote") always quotes specified strings with double quotation marks, for external process invocation. But some problematic applications can't work correctly, when command line arguments are quoted: see issue4463 for detail. On the other hand, quoting itself is needed to specify arguments containing whitespaces and/or some special characters exactly. This patch makes "windows.shellquote" examine the specified string and quote it only when it may have to be quoted for safety.

extdiff: rename the name of an argument for readability To reduce amount of changes for review-ability, previous patch uses "args" as argument name of "dodiff()", even though "args" includes also the name of command to be executed (or full-path of it). This patch replaces "args" by more appropriate name "cmdline".

extdiff: avoid unexpected quoting arguments for external tools (issue4463) Before this patch, all command line arguments for external tools are quoted by the combination of "shlex.split" and "util.shellquote". But this causes some problems. - some problematic commands can't work correctly with quoted arguments For example, 'WinMerge /r ....' is OK, but 'WinMerge "/r" ....' is NG. See also below for detail about this problem. https://bitbucket.org/tortoisehg/thg/issue/3978/ - quoting itself may change semantics of arguments For example, when the environment variable CONCAT="foo bar baz': - mydiff $CONCAT => mydiff foo bar baz (taking 3 arguments) - mydiff "$CONCAT" => mydiff "foo bar baz" (taking only 1 argument) For another example, single quoting (= "util.shellquote") on POSIX environment prevents shells from expanding environment variables, tilde, and so on: - mydiff "$HOME" => mydiff /home/foobar - mydiff '$HOME' => mydiff $HOME - "shlex.split" can't handle some special characters correctly It just splits specified command line by whitespaces. For example, "echo foo;echo bar" is split into ["echo", "foo;echo", "bar"]. On the other hand, if quoting itself is omitted, users can't specify options including space characters with "--option" at runtime. The root cause of this issue is that "shlex.split + util.shellquote" combination loses whether users really want to quote each command line elements or not, even though these can be quoted arbitrarily in configurations. To resolve this problem, this patch does: - prevent configurations from being processed by "shlex.split" and "util.shellquote" only (possibly) "findexe"-ed or "findexternaltool"-ed command path is "util.shellquote", because it may contain whitespaces. - quote options specified by "--option" via command line at runtime This patch also makes "dodiff()" take only one "args" argument instead of "diffcmd" and "diffopts. It also omits applying "util.shellquote" on "args", because "args" should be already stringified in "extdiff()" and "mydiff()". The last hunk for "test-extdiff.t" replaces two whitespaces by single whitespace, because change of "' '.join()" logic causes omitting redundant whitespaces.

subrepo: add forgotten annotation for reverting git subrepos Support for reverting git subrepos was added earlier, but the annotation to handle any subrepo errors was forgotten.

subrepo: add full revert support for git subrepos Previously, revert was only possible if the '--no-backup' switch was specified. Now, to support backups, we explicitly go over all modified files in the subrepo.

setup: don't fail when Python doesn't have the cygwinccompiler package Some Python installations like the ones available from the optware project for the Synology DiskStation NASes don't have that package, which means running the setup script will crash and exit right away. Instead, we now just use an empty/fake class for the HackedMingw32CCompiler, which we likely won't use anyway.

tests: adapt glob pattern to fix test with NetBSD's sh(1) (issue4484)

tests: run 'cvs init' only on non-existent directories (issue4482) Do not create cvsroot directory since cvs-1.12 errors out if CVSROOT exists before 'cvs init'.

remove: use vfs instead of os.path + match.rel() for filesystem checks

forget: use vfs instead of os.path + match.rel() for filesystem checks

tests: make a multi-statement hook in bundle2-exchange Windows compatible This is similar to the fix in 7dd1870120b2.

sshpeer: more thorough shell quoting This fixes an issue spotted by Jesse Hertz.

merge with stable

i18n-ja: synchronized with 3b84bde06d17

i18n-pt_BR: synchronized with 53a65929ef1f

localrepo: use the vfs join method to implement the localrepo join method This will make it possible to customize the behavior of the join method by changing the vfs class (e.g. by using the altvfs" class introduced recently). Note that we could have modified the VFS join methods to acept a set of optional paths in the same way thta the localrepo join method does. However it seemed simpler to simply call os.path.join before calling self.vfs.join.

localrepo: introduce shared method to check if a repository is shared Up until now we compared the "path" and "sharedpath" repository properties to check if a repository is shared. This was relying an implementation detail of shared repositories. In order to make it possible to change the way shared repositories are implemented, we encapsulate this check into its own localrepo method, called shared. This new method returns None if the repository is shared, and something else (for now a string describing the short of share) otherwise. The reason why I did not call this method "isshared" and made it return a boolean is that I plan to introduce a new type of shared repository soon. # NOTES: This is the first patch in a series whose purpose is to add support for creating "full repository shares", which are repositories that share everything with the repository source except their current revision, branch and bookmark. This series is RFC because I am not very sure of some of the solutions I took. Comments are welcome!

trydiff: use 'ctx1.flags()' for symmetry with 'ctx2.flags()'

trydiff: use 'not in addedset' for symmetry with 'not in removedset' With the previous change in place, we can safely use 'addedset'.

trydiff: simplify checking for additions In the body of the loop in trydiff(), there are conditions like: addedset or (f in modifiedset and to is None) The second half of that expression is to account for the fact that merge-in additions appear as additions. By instead fixing up the sets of modified and added files to compensate for this fact, we can simplify the body of the loop. It also fixes one case where the addedset was checked without the additional check (the "have we already reported a copy above?" case in the code, also see fixed test case). The similar condition with 'removedset' in it seems to have served no purpose even before this change, so it could have been simplified even before.