transaction: support for callbacks during abort Previous transaction work added callbacks to be called during regular transaction commit/close. As part of refactoring Mozilla's pushlog extension (an extension that opens a SQLite database and tries to tie its transaction semantics to Mercurial's transaction), I discovered that the new transaction APIs were insufficient to avoid monkeypatching transaction instance internals. Adding a callback that is called during transaction abort removes the necessity for monkeypatching and completes the API.

debugnamecomplete: use new name api Instead of hardcoding a list of places to check, we use the new repo.names api to get a list of potential names to complete.

debugnamecomplete: rename from debuglabelcomplete Now that we have decided on the use of 'name' instead of 'label' we rename this function accordingly. The old method 'debuglabelcomplete' has been left as a deprecated command so that current scripts don't break.

namespaces: add __iter__ and iteritems methods Iterating over all the namespaces is a common operation, naturally, so we add those methods now. Since we are using a sorted dictionary, this method just calls the underlying __iter__ or iteritems method.

namespaces: add 'listnames' property Currently, we have no way to list all the names in a given namespace. This is needed for things such as tab completion. Future patches will use this patch for exactly that purpose.

test-bundle2-exchange: create temp script in $TESTTMP, not $TESTDIR The bundle2-pushkey-hook.sh script is currently created in $TESTTMP, and leaves an untracked file in that directory (tests/) after running.

manifest: drop withflags() method, which is now unused

context: use new manifest.diff(clean=True) support This further simplifies the status code. This simplification comes at a slight performance cost for `hg export`. Before, on mozilla-central: perfmanifest tip ! wall 0.265977 comb 0.260000 user 0.240000 sys 0.020000 (best of 38) perftags ! result: 162 ! wall 0.007172 comb 0.010000 user 0.000000 sys 0.010000 (best of 403) perfstatus ! wall 0.422302 comb 0.420000 user 0.260000 sys 0.160000 (best of 24) hgperf export tip ! wall 0.148706 comb 0.150000 user 0.150000 sys 0.000000 (best of 65) after, same repo: perfmanifest tip ! wall 0.267143 comb 0.270000 user 0.250000 sys 0.020000 (best of 37) perftags ! result: 162 ! wall 0.006943 comb 0.010000 user 0.000000 sys 0.010000 (best of 397) perfstatus ! wall 0.411198 comb 0.410000 user 0.260000 sys 0.150000 (best of 24) hgperf export tip ! wall 0.173229 comb 0.170000 user 0.170000 sys 0.000000 (best of 55) The next set of patches introduces a new manifest type implemented almost entirely in C, and more than makes up for the performance hit incurred in this change.

manifest: add optional recording of clean entries to diff This makes manifest slightly easier to use for status code.

context: use manifest.diff() to compute most of status We can do a little tiny bit better by enhancing manifest.diff to optionally include files that are in both sides. This will be done in a followup patch.

trydiff: replace dodiff=True/'binary' by binarydiff=False/True

trydiff: replace 'dodiff = False' by 'continue' The 'dodiff' variable is initialized to True and may later be set to either False or "binary". When it's set to False, we skip everything after that point, so we can simplify by instead continue-ing (the loop). We can then also drop the 'if dodiff', since it will always be true.

trydiff: make addindexmeta() unconditionally add index meta The conditional-ness is not clear from the name and there is only one caller, so it's clearer to check on the call site. Moving it also makes addindexmeta() no longer close on the 'opts' variable.

trydiff: remove unused code for git index of "combined diff" We only ever produce diffs with one pre- and one post-image, so remove the code for displaying "index" lines for combined diffs.

diff: remove dead assignment

localrepo: show headline notes in commitctx before showing filenames commitctx already showed notes with filenames but didn't provide any context. It is just as relevant to know when manifest or changelog is committed. So, in addition to filenames, also show headlines 'committing files:', 'committing manifest' and 'committing changelog'.

bundle: when verbose, show what takes up the space in the generated bundle This is kind of similar to the debugbundle command but gives summarized actual uncompressed number of bytes when creating the bundle. The numbers are as usable as the bundle format is efficient. Hopefully bundle2 will make it a better indicator of actual entropy. This is useful when accepting pull requests to assess whether the repo size increase seems reasonable for the diff before pushing stuff upstream, It has helped me catching large files that should have been committed as largefiles but was committed as regular files in intermediate changesets. This output doesn't combine well with debug output so we only enable it when verbose without debug.

setdiscovery: factorize similar sampling code We are using full sampling of 'fullsamplesize' in both case. The only difference is the debug message. So we factorise the sampling code and put the message in an extra conditional. This is going to help making changes around the sampling logic. Such changes are needed to improve discovery performance on highly headed repository.

setdiscovery: drop shadowed 'undecided' assignment The 'undecided' variable was never used before being overwritten a few lines later.

hgweb: extract changelist entry generation into own function I want to supplement changelist entries (used by shortlog and changelog endpoints) with custom metadata from an extension. i.e. I have extra per-changeset metadata that I wish to make available to templates so it can be rendered on hgweb. To facilitate this, I've extracted the logic for creating a changeset data structure into its own function, where it can be wrapped by extensions. Ideally, hgweb would use the same templater as the command line and have full access to templatekw.keywords. But that's a lot of work. This patch gets us some of the benefit without all the work. Many other hgweb commands could benefit from similar refactorings. I'm going to wait to see how this patch is received before I go crazy on extracting inline functions.

templates: use CSS classes for diff styling Use of inline style for diff styling led to significant browser memory usage on large diffs. Moving the styling into CSS classes corrects this. This patch is based on work from https://bugzilla.mozilla.org/show_bug.cgi?id=766952 and https://hg.mozilla.org/hgcustom/version-control-tools/rev/2c355a580af6

templates: remove unnecessary <span> The <span> on diffline was useless. It was only making browsers work harder. Remove it.

revset: introduce new operator "##" to concatenate strings/symbols at runtime Before this patch, there is no way to concatenate strings at runtime. For example, to search for the issue ID "1234" in descriptions against all of "issue 1234", "issue:1234", issue1234" and "bug(1234)" patterns, the revset below should be written fully from scratch for each issue ID. grep(r"\bissue[ :]?1234\b|\bbug\(1234\)") This patch introduces new infix operator "##" to concatenate strings/symbols at runtime. Operator symbol "##" comes from the same one of C pre-processor. This concatenation allows parametrizing a part of strings in revset queries. In the case of example above, the definition of the revset alias using operator "##" below can search issue ID "1234" in complicated patterns by "issue(1234)" simply: issue($1) = grep(r"\bissue[ :]?" ## $1 ## r"\b|\bbug\(" ## $1 ## r"\)") "##" operator does: - concatenate not only strings but also symbols into the string Exact distinction between strings and symbols seems not to be convenience, because it is tiresome for users (and "revset.getstring" treats both similarly) For example of revset alias "issue()", "issue(1234)" is easier than "issue('1234')". - have higher priority than any other prefix, infix and postfix operators (like as "##" of C pre-processor) This patch (re-)assigns the priority 20 to "##", and 21 to "(", because priority 19 is already assigned to "-" as prefix "negate".

largefiles: pass a matcher instead of a raw file list to removelargefiles() This is consistent with addlargefiles(), and will make it easier to get the paths that are printed correct when recursing into subrepos or invoking from outside the repository. It also now restricts the path that the addremove is performed on if a path is given, as is done with normal files. The repo.status() call needs to exclude clean files when performing an addremove, because the addremove override method calling this used to pass the list of files to delete, which caused the matcher to only consider those files in building the status list. Now the matcher is restricted only to the extent that the caller requested- usually directories if at all. There's no reason for addremove to care about clean files anyway- we don't want them deleted.

hgweb: allow viewing diffs against p1 or p2 for merge commits (issue3904) This adds UI portion of the feature that has resided in mercurial since 2012. Back then the interface was added together with the code, but was shortly backed out because it was deemed "not ready". Code, however, stayed. For the original feature and its implementation, see issue2810 and d605a82cf189. In short, the backed-out interface had two outstanding issues: 1. it was introducing an entirely new term (baseline) and 2. it was present on every changeset's page, even for changesets with 1 parent (or no parents), which didn't make sense This patch implements a hopefully better interface because: 1. it uses the usual terms (diff) and 2. it only shows up when there actually are 2 parents.

namespaces: remove templatename method on the namespaces object Now that there is an object for each individual namespace, we use the templatename property on that object instead of the method on the collection of namespaces.

namespaces: remove names method on the namespaces object Now that there is an object for each individual namespace, we use that instead of the method on the collection of namespaces.

templatekw: update namespace calls Previous patches changed the namespace api to be more of an object-oriented approach. This patch updates the template function to use said api changes.

namespaces: add __getitem__ property Since the namespaces object uses an underlying (sorted) dictionary to store the namespaces, it makes sense to expose this to naturally gain access to those namespaces.

merge with stable

cmdutil.jsonchangeset: properly compute added and removed files jsonchangeset._show() was computing the reverse status of the current changeset. As a result, added files were showing up as removed and removed files were showing up as adds. There were existing tests for this code and they were flat out wrong.

largefiles: convert addlargefiles() to vfs

rebase: clarify comment about merge ancestor when rebasing merges The code for picking a merge ancestor when rebasing merges had a long and incorrect comment. The comment would perhaps have been fine as commit message but does not make the code more readable or maintainable and is a bad substitute for correct and readable code. The correct essense of the comment is quite trivial: a merge of an ancestor of the rebase destination and an 'outside' revision can be rebased as if it was a linear change, using 'destination ancestor parent' as base and pretty much ignoring the 'outside' revision. The code path where the comment is placed is however also used for other kinds of merge rebases. The comment is thus not really correct and not helpful. I think it would be better to drop the comment and rewrite the code.

status: don't list files as both clean and deleted Tracked files that are deleted should always be reported as such, no matter what their state was in earlier revisions. This is encoded in in two conditions in the loop in basectx._buildstatus() for modified and added files, but the check is missing for clean files. We should check for clean files too, but instead of adding the check in a third place, move it earlier and skip most of the loop body for deleted files.

status: don't list files as both removed and deleted When calculating status involving the working copy and a revision other than the parent of the working copy, the files that are not in the working context manifest ('mf2' in the basectx._buildstatus()) will be reported as removed (note that deleted files _are_ in the working context manifest). However, if the file is reported as deleted in the dirstate, it will get that status too (as shown by failing tests). Fix by removing deleted files from the 'removed' list after the main loop in _buildstatus().

revset-filelog: handle hidden linkrev for file missing for head (issue4490) The fix for linkrev pointing to hidden revision was crashing when the file was missing from head's manifest. We now properly handle this case. (yes I feel silly)

run-tests: automatically add (glob) to "saved backup bundle to" lines Avoid spending too much time adding (glob) after running run-tests -i. This doesn't handle all cases but it helps. The run-tests tests add a bit of escaping of trailing (glob) in the output to avoid interference from the outer test runner. The regexp for matching the output lines contains a group for making multiline substitute in a way that works with Python before 2.7.

transaction: use the right location when cleaning up backup file (issue4479) The location variable fetch from the loop and the one used to actually fetch it mismatched. We fix the name to ensure file outside of store are cleaned up.

merge with stable

revset: delay showing parse error for the revset alias until it is referred Before this patch, a problematic revset alias aborts execution immediately, even if it isn't referred in the specified revset. If old "hg" may be used too (for example, bisecting Mercurial itself), it is also difficult to write alias definitions using features newly introduced by newer "hg" into configuration files, because such alias definitions cause unexpected abortion at parsing revset aliases with old "hg". This patch delays showing parse error for the revset alias until it is actually referred at runtime. This patch detects referring problematic aliases in "_expandaliases" by examination of "revsetalias.error", which is initialized with the error message only when parsing fails. For usability, this patch also warns about problematic aliases, even if they aren't referred at runtime. This should help users to know potential problems in their alias definitions earlier.

revset: drop pre-lazyset optimization for stringset of subset == entire repo It was introduced at e44ebd2a142a, where spanset.__contains__() did not exist. Nowadays, we have to pay huge penalty for len(subset). The following example showed that OR operation could be O(n * m^2) (n: len(repo), m: number of OR operators, m >= 2) probably because of filteredset.__len__. revset #0: 0|1|2|3|4|5|6|7|8|9 0) wall 8.092713 comb 8.090000 user 8.090000 sys 0.000000 (best of 3) 1) wall 0.445354 comb 0.450000 user 0.430000 sys 0.020000 (best of 22) 2) wall 0.000389 comb 0.000000 user 0.000000 sys 0.000000 (best of 7347) (0: 3.2.4, 1: 3.1.2, 2: this patch)

keyword: update copyright year

keyword: use vfs.reljoin and util.unlinkpath to remove kwdemo

largefiles: properly sync lfdirstate after removing largefiles The more aggressive synchronization of lfdirstate that was backed out in 1265a3a71d75 masked the problem where lfdirstate would hold an 'R' for a largefile that was added and then removed without a commit between. We could just conditionally call lfdirstate.drop() or lfdirstate.remove() here, but this also properly updates lfdirstate if the standin doesn't exist for the file somehow (i.e. call drop instead of remove). Without this change, the precommit status in the commit command immediately after the test change lists the removed (and never committed) largefile as 'R'. It can also lead to situations where the status command reports the same, long after the commit [1]. [1] http://www.selenic.com/pipermail/mercurial-devel/2015-January/065153.html

linkrev-filelog: handle filtered linkrev with no visible children (issue4307) If the file revision with a filtered linkrev does not have any (unfiltered) children, we cannot use it to bound the search for another introduction. Instead, we have to look at the file revision used by each head changeset. If one of them uses this file revision, we know there is another occurrence and we have a starting point. See inline comments for details. Adding some kind of permanent reference of all the introductions of a file revision instead of just the first one would be much better. But this is more difficult. I hope to take that into account in the next repository format.

linkrev: work around linkrev to filtered entry in 'filelog' revset This revset is used by 'hg log FILENAME'. This prevent bugs when used on a repository with hidden revisions. Instead of just discarding file revisions whose linkrevs point to filtered revisions, we put them aside and post-process them trying to find a non-filtered introduction. See inline documentation for details about how it works. This only fixes some of the problems. Once again, more will be needed when we can cannot rely on child revisions of a file to find linkrev-shadowned revisions. A test is added for 'hg log' catching such cases.

namespaces: update documentation and code indentation The previous patch changed just the functionality, while this patch focuses on the documentation and indentation to keep review simple.

namespaces: use namespace object instead of dictionary This isn't as bad as the diff seems, it only looks like scary. In this patch, we use the 'namespace' object instead of accessing keys in a dictionary. This required the 'templatename' and 'names' method to change their implementation. Later, we will remove these functions entirely due to a better api.

namespaces: copy implementation to new namespace object

namespaces: add a namespace object Currently, we use a dictionary object to store the namespace properties. This is python so let's use an object. This will allow us to be more flexible in our method signatures in the future.

localrepo: use vfs.reljoin rather than os.path.join in the localrepository class The removes the last remaining usage of os.path.join inside teh localrepository class.

largefiles: drop a redundant check for patterns when removing without --after This is in the way of passing a matcher to removelargefiles(). This method is called in exactly two places- first in overrides.addremove() (but only if the pattern list passed to it is not empty), and second in the commands.remove() override. But since the latter calls commands.remove() first, which also does this check, it isn't needed here.

context: override _dirstatestatus in workingcommitctx for correct matching Before this patch, the result of "status()" on "workingcommitctx" may incorrectly contain files other than ones to be committed, because "workingctx._dirstatestatus()" returns the result of "dirstate.status()" directly. For correct matching, this patch overrides "_dirstatestatus" in "workingcommitctx" and makes it return matched files only in "self._status". This patch uses empty list for "deleted", "unknown" and "ignored" of status, because status between "changectx"s also makes them empty.

context: avoid breaking already fixed self._status at ctx.status() Before this patch, "status()" on "workingcommitctx" with "always match" object causes breaking "self._status" in "workingctx._buildstatus()", because "workingctx._buildstatus()" caches the result of "dirstate.status()" into "self._status" for efficiency, even though it should be fixed at construction for committing. For example, template function "diff()" without any patterns in "committemplate" implies "status()" on "workingcommitctx" with "always match" object, via "basectx.diff()" and "patch.diff()". Then, broken "self._status" causes committing unexpected files. To avoid breaking already fixed "self._status" at "ctx.status()", this patch overrides "_buildstatus" in "workingcommitctx". This patch doesn't write out the result of template function "diff()" in "committemplate" in "test-commit.t", because matching against files to be committed still has an issue fixed in subsequent patch.

context: add workingcommitctx for exact context to be committed Before this patch, "workingctx" is also used for the context to be committed. But "workingctx" works incorrectly in some cases. For example, even when only some of changed files in the working directory are committed, "status()" on "workingctx" object for committing recognizes files not to be committed as changed, too. As the preparation for fixing these issues, this patch chooses adding new class "workingcommitctx" for exact context to be committed, because switching by the flag (like "self._fixedstatus" or so) in some code paths of "workingctx" is less readable and maintenancable.

context: make unknown/ignored/clean of cached status empty for equivalence Before this patch, "workingctx.status" caches the result of "dirstate.status" directly into "self._status". But "dirstate.status" is invoked with False "list*" arguments in normal "self._status" accessing route, and this makes "unknown"/"ignored"/"clean" of status empty. This may cause unexpected result of code paths internally accessing to them (accessors for external usage are already removed by previous patch). This patch makes "unknown"/"ignored"/"clean" of cached status empty for equivalence. Making them empty is executed only when at least one of "unknown", "ignored" or "clean" has files, for efficiency.

templatefilters.json: stabilize output The json filter was previously iterating over keys in an object in an undefined order. Let's throw a sorted() in there so output is consistent. It's somewhat frightening that there are no tests for the json filter. Subsequent commits will add them, so we pass on the opportunity to add them here.

templatefilters.json: call functions The "changeset" template from hgweb is using a lambda in the "diffsummary" key. In preparation for enabling JSON output from hgweb, teach the json filter how to call functions.

merge with stable

linkrev: also adjust linkrev when bootstrapping annotate (issue4305) The annotate logic now use the new 'introrev' method to bootstrap its traversal. This catches issues from linkrev-shadowing of the changeset introducing the version of a file in source changeset. More tests have been added to display pathological cases.