add: pass options via keyword args The largefiles extensions needs to be able to pass --large, --normal and --lfsize to subrepos via cmdutil.add() and hgsubrepo.add(). Rather than add additional special purpose arguments, stop extracting the existing args from the **opts passed to commands.add() and just pass them along.

largefiles: don't pop largefile-specific arguments to the add command The arguments will need to stay present when making add work with subrepos.

share: replace the bookmarks.shared file with an entry on a new "shared" file cd79fb4d75fd introduced a way to share bookmarks. When a repository share that shares bookmarks was created, a .hg/bookmarks.shared file was created to mark the repository share as one that shares its bookmarks. We have plans to introduce other levels of sharing, including a "full share" mode. Rather than creating a new ".shared" file for each new thing that we may want to share It seems better to create a single "shared" file that will list what is shared for a given shared repository. This should make it much easier to get a list of everything that is shared by a given shared repository. The shared file contains a list of shared "items" (such as bookmarks). Each shared "item" is added as a new line in the file. For now the only possible entry in the file is "bookmarks".

docker: support Fedora 21

rpm: make Python 2.7.9 the default Python to include in rpms for EL 5 Use the new and more TLS support in Python 2.7.9.

contrib: make Python 2.7.9 the default in Makefile.python We should utilize (and test) the big API changes and new TLS functionality in Python 2.7.9 whenever possible.

localrepo: remove all external users of localrepo.wopener This change touches every module in which repository.wopener was being used, and changes it for the equivalent repository.wvfs. It should now be possible to remove localrepo.wopener.

localrepo: remove all external users of localrepo.sopener This change touches every module in which repository.sopener was being used, and changes it for the equivalent repository.svfs. It should now be possible to remove localrepo.sopener.

localrepo: remove all external users of localrepo.opener This change touches every module in which repository.opener was being used, and changes it for the equivalent repository.vfs. This is meant to make it easier to split the repository.vfs into several separate vfs. It should now be possible to remove localrepo.opener.

log: use namespace logname and colorname Now that we have the machinary to change the log name and the color label used, let's use that. Tests have been updated accordingly.

namespaces: add colorname member to namespace object Previously, there was no way to change the color label used for 'hg log' output. This patch just adds the member to the object, a future patch will change 'hg log' to use this.

namespaces: add logname member to namespace object Previously, there was no way to change the name used for 'hg log' output. This was inconvenient for extensions that want a template name longer than 12 characters (e.g. remotebookmarks) but a different name for 'hg log'. This patch only adds the member to the object, a future patch will update the 'hg log' code.

namespaces: use named args for namespace api This is just a style change but makes adding new arguments more robust for callers.

namespaces: make the constructor into named args None of the arguments are truly optional but this makes adding future arguments more robust and perhaps optional.

dispatch: only check compatibility against major and minor versions (BC) Extensions can declare compatibility with Mercurial versions. If an error occurs, Mercurial will attempt to pin blame on an extension that isn't marked as compatible. While all bets are off when it comes to the internal API, my experience has shown that a monthly/patch release of Mercurial has never broken any of the extensions I've written. I think that expecting extensions to declare compatibility with every patch release of Mercurial is asking a bit much and adds little to no value. This patch changes the blame logic from exact version matching to only match on the major and minor Mercurial versions. This means that extensions only need to mark themselves as compatible with the major, quarterly releases and not the monthly ones in order to stay current and avoid what is almost certainly unfair blame. This will mean less work for extension authors and almost certainly fewer false positives in the blame attribution.

test-extension: use a realistic Mercurial version Mercurial doesn't define the 3rd "patch" version field for major releases. Don't use it in tests.

test-extension: improve test readability There was a mountain of text in this file that made reading the tests difficult. Inserting some line breaks greatly improves the situation.

bundle2: enforce parttype as alphanumerical The binary format description has always stated that the parttype should be simple, but it was never really enforced. Recent discussions have convinced me we want to keep the part type simple and easy to debug. There is enough extensibility in the rest of the format.

tests: conditionalize test-tools.t for Windows support I still get the following diff on Windows 7 with NTFS. I'm not sure if it is expected and we should wildcard the link count, or if something needs to be fixed. @@ -58,7 +58,7 @@ $ ln bar baz $ f bar -n baz -l --hexdump -t --sha1 --lines=9 -B 20 - bar: file, links=2, newer than baz, sha1=612ca68d0305c821750a + bar: file, links=0, newer than baz, sha1=612ca68d0305c821750a\r (esc) 0000: 31 0a 32 0a 33 0a 34 0a 35 0a 36 0a 37 0a 38 0a |1.2.3.4.5.6.7.8.| 0010: 39 0a |9.|

dirstate: ignore negative debug.dirstate.delaywrite values - they crashed it Sleep can only travel forward in time, not back.

linkrev: use the right manifest content when adjusting linrev (issue4499) When the manifest revision is stored as a delta against a non-parent revision, '_adjustlinkrev' could miss some file update because it was using the delta only. We now use the 'fastread' method that uses the delta only when it makes sense. A test showcasing on the of possible issue have been added.

unpacker: fix missing arg for py2.4

branchcache: make _rbcrevslen handling more safe self._rbcrevslen is used to keep track of the number of good records on disk. It should thus not be updated before the records actually have been written to disk.

branchcache: add debug output whenever cache files use truncate The cache files are usually append only but will automatically be truncated and recover in exceptional situations. Add a debug notice when such exceptional situations are encountered.

tests: rework revision branch cache tests The tests that were added with the revision branch cache in 678f53865c68 had suffered from bit rot in the development iterations. They were no longer that "good". Now, the tests are rewritten and reworked to be more readable and maintainable and relevant for the actual implementation. This also utilizes the new 'f' helper tool for keeping an eye the cache files.

tests: add 'f' tool for cross platform file operations in the tests This tool is like the collection of tools found in a unix environment but are cross platform and stable and suitable for our needs in the test suite. The main reason it is "needed" now is for hexdump of revision branch cache to keep an eye on how it changes and make sure the format is stable. It is a very generic tool that can end up being used a lot in tests, so I gave it very generic name.

run-tests: include testdir in $PATH so tests easily can use helper tools The testdir is already added to $PYTHONPATH - I think it makes sense and is convenient to add it to $PATH too. The following binaries are invoked from tests using full path with $TESTDIR/ - they can now be used without specifying path: dumbhttp.py dummyssh filterpyflakes.py generate-working-copy-states.py get-with-headers.py hghave histedit-helpers.sh killdaemons.py md5sum.py notcapable printenv.py readlink.py revlog-formatv0.py run-tests.py svn-safe-append.py svnxml.py tinyproxy.py

filelog: use censored revlog flag bit to quickly check if a node is censored

revlog: verify censored flag when hashing added revision fulltext When receiving a delta via exchange, three possible storage outcomes emerge: 1. The delta is added directly to the revlog. ("fast-path") 2. A freshly-computed delta with a different base is stored. 3. The new revision's fulltext is computed and stored outright. Both (2) and (3) require materializing the full text of the new revision by applying the delta to its base. This is typically followed by a hash check. The new flags argument allows callers to _addrevision to signal that they expect that hash check to fail. We can use this opportunity to verify that expectation. If the hash fails, require the flag be set; if the hash passes, require the flag be unset. Rather than simply eliding the hash check, this approach provides some assurance that the censored flag is not applied to valid revisions. Read more at: http://mercurial.selenic.com/wiki/CensorPlan

revlog: add flags argument to _addrevision, update callers use default flags For revlog index flags to be useful to other parts of Mercurial, they need to be settable when writing revisions. The current use case for revlog index flags is the censorship feature: http://mercurial.selenic.com/wiki/CensorPlan While the censor flag could be inferred in _addrevision by interrogating the text/delta being added, that would bury the censorship logic and inappropriately couple it to all revision creation.

revlog: define censored flag for revlogng index This flag bit will be used to cheaply signal censorship presence to upper layers (exchange, verify). It indicates that censorship metadata is present but does not attest to the verifiability of that metadata. For the censorship design, see: http://mercurial.selenic.com/wiki/CensorPlan

localrepo: remove all internal uses of localrepo.wopener It has been replaced with localrepo.wvfs.

localrepo: remove all internal uses of localrepo.sopener It has been replaced with localrepo.svfs.

localrepo: remove all internal uses of localrepo.opener It has been replaced with localrepo.vfs. In the future we may split the vfs into different vfs objects to access different elements of the repository.

sslutil: drop defunct ssl version constants Nobody outside sslutil should be using these constants anyway.

sslutil: use saner TLS settings on Python 2.7.9 Asking for TLSv1 locks us out of TLSv1_2 etc. This is at least less bad. Ideally we'd use ssl.create_default_context(), but that causes more mayhem in the testsuite than I really want to deal with right now.

sslutil: drop support for clients of sslutil specifying a TLS version We really just want to support the newest thing possible, so we may as well consolidate that knowledge into this module. Right now this doesn't change any behavior, but a future change will fix the defaults for Python 2.7.9 so we can use slightly better defaults there (which is the only place it's possible at the moment.)

discovery: run discovery on filtered repository We have been running discovery on unfiltered repository for quite some time. This was aimed at two things: - save some bandwith by prevent the repushing of common but hidden changesets - allow phases changes on secret/hidden changeset on bare push. The cost of this unfiltered discovery combined with evolution is actually really high. Evolution likely create thousand of hidden heads, and the discovery is going to try to discovery if each of them are common or not. For example, pushing from my development mercurial repository implies 17 discovery round-trip. The benefit are rare corner cases while the drawback are massive. So we run the discovery on a filtered repository again. We add some hack to detect remote heads that are known locally and adds them to the common set anyway, so the good behavior of most of the corner case should remains. But this will not work in all cases. This bring my discovery phase back from 17 round-trips to 1 or 2.

revset: check for collisions between alias argument names in the declaration Before this patch, collisions between alias argument names in the declaration are ignored, and this silently causes unexpected alias evaluation. This patch checks for such collisions, and aborts (or shows a warning) when collisions are detected. This patch doesn't add a test to "test-revset.t", because a doctest is enough to test the collisions detection itself.

revset: parse alias declaration strictly by _parsealiasdecl Before this patch, alias declaration is parsed by string base operations: matching against "^([^(]+)\(([^)]+)\)$" and splitting by ",". This overlooks many syntax errors like below (see the previous patch introducing "_parsealiasdecl" for detail): - un-closed parenthesis causes being treated as "alias symbol" - symbol/function name aren't examined whether they are valid or not - invalid argument list causes unexpected argument names To parse alias declaration strictly, this patch replaces parsing implementation by "_parsealiasdecl". This patch tests only one typical declaration error case, because error detection itself is already tested in the doctest of "_parsealiasdecl". This also removes class property "args" and "error", because these are certainly initialized in "revsetalias.__init__".

revset: introduce "_parsealiasdecl" to parse alias declarations strictly This patch introduces "_parsealiasdecl" to parse alias declarations strictly. For example, "_parsealiasdecl" can detect problems below, which current implementation can't. - un-closed parenthesis causes being treated as "alias symbol" because all of declarations not in "func(....)" style are recognized as "alias symbol". for example, "foo($1, $2" is treated as the alias symbol. - alias symbol/function names aren't examined whether they are valid as symbol or not for example, "foo bar" can be treated as the alias symbol, but of course such invalid symbol can't be referred in revset. - just splitting argument list by "," causes overlooking syntax problems in the declaration for example, all of invalid declarations below are overlooked: - foo("bar") => taking one argument named as '"bar"' - foo("unclosed) => taking one argument named as '"unclosed' - foo(bar::baz) => taking one argument named as 'bar::baz' - foo(bar($1)) => taking one argument named as 'bar($1)' To decrease complication of patch, current implementation for alias declarations is replaced by "_parsealiasdecl" in the subsequent patch. This patch just introduces it. This patch defines "_parsealiasdecl" not as a method of "revsetalias" class but as a one of "revset" module, because of ease of testing by doctest. This patch factors some helper functions for "tree" out, because: - direct accessing like "if tree[0] == 'func' and len(tree) > 1" decreases readability - subsequent patch (and also existing code paths, in the future) can use them for readability This patch also factors "_tokenizealias" out, because it can be used also for parsing alias definitions strictly.

revset: store full detail into revsetalias.error for error source distinction Before this patch, any errors in the declaration of revset alias aren't detected at all, and there is no information about error source in the error message. As a part of preparation for parsing alias declarations and definitions more strictly, this patch stores full detail into "revsetalias.error" for error source distinction. This makes raising "Abort" and warning potential errors just use "revsetalias.error" without any message composing.

revset: factor out composing error message for ParseError to reuse This patch defines the composing function not in "ParseError" class but in "revset" module, because: - "_()" shouldn't be used in "ParseError", to avoid adding "from i18n import _" i18n" to "error" module - generalizing message composition of"ParseError" for all code paths other than revset isn't the purpose of this patch we should also take care of showing "unexpected leading whitespace" for some code paths, to generalize widely.

revset: make tokenize extensible to parse alias declarations and definitions Before this patch, "tokenize" doesn't recognize the symbol starting with "$" as a valid one. This prevents revset alias declarations and definitions from being parsed with "tokenize", because "$" may be used as the initial letter of alias arguments. BTW, the alias argument name doesn't require leading "$" itself, in fact. But we have to assume that users may use "$" as the initial letter of argument names in their aliases, because examples in "hg help revsets" uses such names for a long time. To make "tokenize" extensible to parse alias declarations and definitions, this patch introduces optional arguments "syminitletters" and "symletters". Giving these sets can change the policy of "valid symbol" in tokenization easily. This patch keeps original examination of letter validity for reviewability, even though there is redundant interchanging between "chr"/"ord" at initialization of "_syminitletters" and "_symletters". At most 256 times examination (per initialization) is cheaper enough than revset evaluation itself. This patch is a part of preparation for parsing alias declarations and definitions more strictly.

largefiles: make linear update set unsure largefiles normal if unchanged 'hg update' would hash all 'unsure' largefiles before performing the merge. It would update the standins but not detect the very common case where the largefile never had been changed by the user but just had been marked with an invalid dirstate mtime to make sure any changes done by the user in the same second would be detected. The largefile would remain in that state and would have to be hashed again next time even though it still not had been changed. Sad trombone. Instead, for largefiles listed as 'unsure' or 'modified', after updating the standin with the actual hash, mark the largefile as normal if it turns out to not be modified relative to the revision in the parent revision. That will prevent it from being hashed again next time.

debugdirstate: don't hide date field with --nodate, just show 'set'/'unset' The value of the dirstate date field cannot be used in tests and we thus have to use debugdirstate with --nodate. It is however still very helpful to be able to see whether the date field has been set or still is unset. The absence of that information made it hard to debug some largefile dirstate issues. This change _could_ make the test suite more unstable ... but that would be places where the test suite or the code should be made more stable. (Note: 'unset' with the magic negative sizes is reliable. 'unset' for normal sizes would probably not be reliable, but there is no such occurrences in the test suite and it should thus be reliable.) This output wastes more horizontal space in the --nodate output, but it also makes things simpler that the output format always is the same. It is just a debug command so let's keep it simple.

debugdirstate: simplify date handling after e7ed5d07cc4c used fixed format

forget: don't report rejected files as forgotten as well It seems like a mistake to report a file as forgotten and rejected. The forgotten list doesn't seem to be used by anything in core, so no test changes.

largefiles: enable subrepo support for forget

namespaces: add revset for 'named(namespace)' This patch adds functionality for listing all changesets in a given namespace via the revset language.

bundles: do not overwrite existing backup bundles (BC) Previously, a backup bundle could overwrite an existing bundle and cause user data loss. For instance, if you have A<-B<-C and strip B, it produces backup bundle B-backup.hg. If you then hg pull -r B B-backup.hg and strip it again, it overwrites the existing B-backup.hg and C is lost. The fix is to add a hash of all the nodes inside that bundle to the filename. Fixed up existing tests and added a new test in test-strip.t

https: support tls sni (server name indication) for https urls (issue3090) SNI is a common way of sharing servers across multiple domains using separate SSL certificates. As of Python 2.7.9 SSLContext has been backported from Python 3. This patch changes sslutil's ssl_wrap_socket to use SSLContext and take a server hostname as and argument. It also changes the url module to make use of this argument. The new code for 2.7.9 achieves it's task by attempting to get the SSLContext object from the ssl module. If this fails the try/except goes back to what was there before with the exception that the ssl_wrap_socket functions take a server_hostname argument that doesn't get used. Assuming the SSLContext exists, the arguments to wrap_socket at the module level are emulated on the SSLContext. The SSLContext is initialized with the specified ssl_version. If certfile is not None load_cert_chain is called with certfile and keyfile. keyfile being None is not a problem, load_cert_chain will simply expect the private key to be in the certificate file. verify_mode is set to cert_reqs. If ca_certs is not None load_verify_locations is called with ca_certs as the cafile. Finally the wrap_socket method of the SSLContext is called with the socket and server hostname. Finally, this fails test-check-commit-hg.t because the "new" function ssl_wrap_socket has underscores in its names and underscores in its arguments. All the underscore identifiers are taken from the other functions and as such can't be changed to match naming conventions.

merge with stable

unpacker: check the right exception type for 2.4

hgweb: fix diffstat links in paper/changeset.tmpl '<a .../>foo</a>' syntax is incorrect, since the first tag just "tries" to close itself and then the actual content follows. It doesn't work, either because web browsers know better than this or because there should be a whitespace before /: '<a />'. So for the hgweb users the links looked normal anyway, but now they are correct in code as well.

hgweb: close <img> elements Templates declare xhtml doctype, which means, in particular, that the document must also be valid xml. So <img> elements must be closed.

hgweb: close <p> elements <p> elements can only contain inline elements, so as soon as browser encounters a block element (e.g. block <div>) "inside" a <p>, it puts an implicit </p>. It's better to do this explicitly.

hgweb: close <th> properly in spartan/filelogentry.tmpl

revset: simplify fullreposet.__and__ to call sort() with boolean flag Note that sort() takes a boolean flag, so other.sort(reverse) was wrong. It just worked fine because there is a top-level function, reverse().

revset: fix spanset.isascending() to honor sort() or reverse() request Because spanset.isascending() ignored the ascending flag, the result of "fullreposet() & x" was always sorted in ascending order. The test case is carefully chosen to call fullreposet.__and__.

hghave: we now support Python 2.7.9's ssl for https

Makefile.python: try curl if wget fails Macs ship with curl and not wget, so this is a nice little tweak for folks testing on OS X.

test-https: glob error messages more so we pass on Python 2.7.9 Python 2.7.9 cleans up how it stringifies SSL errors, so we have to look only for the important bit (certificate verify failed) rather than looking for specific ssl module goop (which is now unstable).

filelog: fix backwards comment for 'backrevref'

filelog: remove trailing "form feed" character

filelog: remove unused variable 'lkr' It's used further down, but it's overwritten before, so it's technically a dead assignment, but unnecessary nevertheless.

branchmap: add seek() to end of file before calling tell() on append open() This is similar to 48c232873a54, which was subsequently modified in 19f5dec2d61f for 2.4. Unexpected test changes on Windows occurred without this.

tests: fix test-casefolding.t output for branchcache This belongs with 7d63398fbfd1. I assume that the failure to read is OK, because there is similar output in test-convert-svn-encoding.t.

setdiscovery: remove '_setupsample' function It is now unused.

setdiscovery: document '_takequicksample'

setdiscovery: drop '_setupsample' usage in '_takequicksample' For '_takefullsample' we can just retrieve the list of head directly and ignore the rest of the complex return values. This was the last call to the infamous '_updatesample' function.

setdiscovery: drop the 'always' argument to '_updatesample' This argument exists because of the complex code flow in '_takequicksample'. It first gets the list of heads and then calls '_updatesample' on an empty initial sample and a size limit matching the differences between the number of heads and the target sample size. Finally the heads and the sample from '_updatesample' were added. To ensure this addition result had the exact target length, the code had to ensure no elements from the heads were added to the '_updatesample' content and therefore was passing this "always included set of heads". Instead we can just update the initial heads sample directly and use the final target size as target size for the update. This removes the need for this 'always' parameter to the '_updatesample' function The test are affected because different set building order results in different random sampling.

setdiscovery: always add exponential sample to the heads As explained in a previous changeset, prioritizing heads too much behaves pathologically when there are more heads than the sample size. To counter this, we always inject exponential samples before reducing to the sample size limit. This already show some benefit in the test themselves, but on a real-world example this moves my discovery for push to pathologically headed repo from 45 rounds to 17 of them. We should maybe ensure that at least 25% of the result sample is heads, but I think the random sampling will be fine in practice.

setdiscovery: directly run '_updatesample' The heads and exponential sample are going to end up in the same set before any extra processing happens. We simplify the code by directly updating a set with heads. Changes in the order the set is built lead to small changes in the random sampling output. But after double checking, I can confirm the input data to the random sampling is consistent.

setdiscovery: stop using '_setupsample' in '_takefullsample' Very few of the return values of '_setupsample' remain in use, so we directly retrieve the value we care about and drop the '_setupsample' call.

setdiscovery: randomly pick between heads and sample when taking full sample Before this changeset, the discovery protocol was too heads-centric. Heads of the undiscovered set were always sent for discovery and any room remaining in the sample were filled with exponential samples (and random ones if any room remained). This behaved extremely poorly when the number of heads exceeded the sample size, because we keep just asking about the existence of heads, then their direct parent and so on. As a result, the 'O(log(len(repo)))' discovery turns into a 'O(len(repo))' one. As a solution we take a random sample of the heads plus exponential samples. This way we ensure some exponential sampling is achieved, bringing back some logarithmic convergence of the discovery again. This patch only applies this principle in one place. More places will be updated in future patches. One test is impacted because the random sample happen to be different. By chance, it helps a bit in this case.

setdiscovery: document the '_updatesample' function This function is central in the sample building process, having it documented help code readability a lot.

setdiscovery: avoid calling any sample building if the undecided set is small If the length of undecided is smaller than the sample size, we can just request information for all of them. This conditional was previously handled by '_setupsample'. But '_setupsample' is in my opinion a problematic function with blurry semantics. Having this conditional explicitly earlier makes the code more explicit and moves us closer to removing this '_setupsample' function.

setdiscovery: delay sample building calls to gather them in a single place Some of the logic around sample building is duplicated in the sample builders, it would clean up thing to extract it in the top function, but this requires all codes to be in the same place. This changeset mostly exists to make the next one more clear.

setdiscovery: drop unused 'initial' argument for '_takequicksample' There is a single call site, and it is always using 'initial=True'. So we just drop the argument and the associated condition.

readmarkers: add a SHA256 fixme note

readmarkers: fast-path single successors and parents This gives about a 5% performance bump.

readmarkers: promote global constants to locals for performance

readmarkers: drop a temporary

readmarkers: read node reading into node length conditional This removes some conditional assignments

readmarkers: drop a temporary Two other temporaries are renamed to fit line-length.

readmarkers: hoist subtraction out of loop comparison

readmarkers: streamline offset tracking This minimizes the number of assignments and operations needed to use offsets.

readmarkers: use unpacker for fixed header

readmarkers: drop metadata temporary

readmarkers: drop date temporary

readmarkers: drop another conditional

readmarkers: drop a conditional

readmarkers: add some whitespace

readmarkers: combine parent conditionals

readmarkers: drop temporary substring assignments Assignments are expensive in inner loops

util: introduce unpacker This allows taking advantage of Python 2.5+'s struct.Struct, which provides a slightly faster unpack due to reusing formats. Sadly, .unpack_from is significantly slower.

perf: add a configurable sleep on startup This is intended to counteract power management by giving a consistent idle period before test runs.

revset: use localrepo revbranchcache for branch name filtering Branch name filtering in revsets was expensive. For every rev it created a changectx and called .branch() which retrieved the branch name from the changelog. Instead, use the revbranchcache. The revbranchcache is used read-only. The revset implementation with generators and callbacks makes it hard to figure out when we are done using/updating the cache and could write it back. It would also be 'tricky' to lock the repo for writing from within a revset execution. Finally, the branchmap update will usually make sure that the cache is updated before any revset can be run. The revbranchcache is used without any locking but is short-lived and used in a tight loop where we can assume that the changelog doesn't change ... or where it not is relevant to us if it does. perfrevset 'branch(mobile)' on mozilla-central. Before: ! wall 10.989637 comb 10.970000 user 10.940000 sys 0.030000 (best of 3) After, no cache: ! wall 7.368656 comb 7.370000 user 7.360000 sys 0.010000 (best of 3) After, with cache: ! wall 0.528098 comb 0.530000 user 0.530000 sys 0.000000 (best of 18) The performance improvement even without cache come from being based on branchinfo on the changelog instead of using ctx.branch(). Some tests are added to verify that the revbranchcache works and keep an eye on when the cache files actually are updated.

branchmap: use revbranchcache when updating branch map The revbranchcache is read on demand before it will be used for updating the branch map. It is written back when the branchmap is written and it will thus use the same locking as branchmap. The revbranchcache instance is short-lived; it is only stored in the branchmap from .update() is invoked and until .write() is invoked. Branchmap already assume that the repo is locked in that case. The use of revbranchcache for branch map updates will make sure that the revbranchcache "always" is kept up-to-date. The perfbranchmap benchmark is somewhat bogus, especially when we can see that the caching makes a significant difference between the realistic case of a first run and the rare case of rerunning it with a full cache. Here are some 'base' numbers on mozilla-central: Before: ! wall 6.912745 comb 6.910000 user 6.840000 sys 0.070000 (best of 3) After - initial, cache is empty: ! wall 7.792569 comb 7.790000 user 7.720000 sys 0.070000 (best of 3) After - cache is full: ! wall 0.879688 comb 0.880000 user 0.870000 sys 0.010000 (best of 4) The overhead when running with empty cache comes from checking, missing and updating it every time. Most of the performance improvement comes from not having to extract the branch info from the changelog. The last doubling of performance comes from no longer having to convert all branch names to local encoding but reuse the few already converted branch names. On the hg repo: Before: ! wall 0.715703 comb 0.710000 user 0.710000 sys 0.000000 (best of 14) After: ! wall 0.105489 comb 0.110000 user 0.110000 sys 0.000000 (best of 87)

branchcache: introduce revbranchcache for caching of revision branch names It is expensive to retrieve the branch name of a revision. Very expensive when creating a changectx and calling .branch() every time - slightly less when using changelog.branchinfo(). Now, to speed things up, provide a way to cache the results on disk in an efficient format. Each branchname is assigned a number, and for each revision we store the number of the corresponding branch name. The branch names are stored in a dedicated file which is strictly append only. Branch names are usually reused across several revisions, and the total list of branch names will thus be so small that it is feasible to read the whole set of names before using the cache. It will however do that it might be more efficient to use the changelog for retrieving the branch info for a single revision. The revision entries are stored in another file. This file is usually append only, but if the repository has been modified, the file will be truncated and the relevant parts rewritten on demand. The entries for each revision are 8 bytes each, and the whole revision file will thus be 1/8 of 00changelog.i. Each revision entry contains the first 4 bytes of the corresponding node hash. This is used as a check sum that always is verified before the entry is used. That check is relatively expensive but it makes sure history modification is detected and handled correctly. It will also detect and handle most revision file corruptions. This is just a cache. A new format can always be introduced if other requirements or ideas make that seem like a good idea. Rebuilding the cache is not really more expensive than it was to run for example 'hg log -b branchname' before this cache was introduced. This new method is still unused but promise to make some operations several times faster once it actually is used. Abandoning Python 2.4 would make it possible to implement this more efficiently by using struct classes and pack_into. The Python code could probably also be micro optimized or it could be implemented very efficiently in C where it would be easy to control the data access.

hgweb: move archive entries outside of <li> in monoblue style archiveentry already includes surrounding <li></li>, so putting archive entries inside <li> element produced incorrect markup.

hgweb: add searchhint to templates/coal/map coal style uses every template (except header.tmpl) directly from paper style, but doesn't use paper/map file. Elements defined in such map files are used in templates as you would expect. For example, paper/search.tmpl contains '{searchhint}' and template engine replaces that with the actual hint. But when coal style reuses paper/search.tmpl, it needs to define searchhint in its map file as well, or template engine will not find it. So let's copy it from paper/map to coal/map. Before this change, if the coal style was selected, the hint for the search field in page header was present, but it was completely empty. Although the absence of searchhint in coal/map produced no error.

largefiles: enable subrepo support for remove Previously, remove failed when operating on a largefile in a subrepo, stating that the file is untracked.

transplant: properly skip empty changeset (issue4423) If resolving a merge conflict result in an empty changesets, we now properly skip the changeset instead of crashing. Original patch from Robert Collins <robertc@robertcollins.net>.

color: add support for colorizing git subrepo diffs

tests: fix test-casefolding.t output This change started with a387b0390082.

synthrepo: new filenames must not also be new directories, and vice-versa When generating many new files into a set of many possible new directories, there is the possibility that the same path is chosen as both file and directory. How likely this is depends on the size of the dictionary used, the generated directory structure and the number of generated files.

status: don't override _buildstatus() in workingcommitctx Now that the caching into _status is done in workingctx._dirstatestatus(), which workingcommitctx._dirstatestatus() does not call, there is no caching to prevent in _buildstatus(), so stop overriding it.

status: cache dirstate status in _dirstatestatus() Since it's only the dirstate status we cache, it makes more sense to cache it in the _dirstatestatus() method. Note that this change means the dirstate status will also be cached when status is requested between the working copy and some other revision, while we currently only cache the result if exactly the status between the working copy and its parent is requested.

localrepo: add ignoremissing parameter to branchtip Previously, in the namespaces api, the only caller of branchtip was singlenode which happened to raise the same exception that branchtip raised: KeyError. This is a minor change but will allow upcoming patches to use repo.branchtip to not raise an exception if a branch doesn't exist. After that, it will be possible for extensions to use the namespace api in a stable way.

namespaces: add method to return a list of nodes for a given name This is a helpful method that some extensions can make use of (e.g. for custom revsets); currently not used in core.

namespaces: add test for log Now that we have enough features in the namespaces api, we add a test for it.

log: use new namespaces api to display names The only caveat here is that branches must be displayed first due to backwards compatibility. The order of namespaces is defined to be the 'update' order which, unfortunately, is not the same as log output order. It's worth mentioning that the log output is still translated the same as before since we are formating our strings the same way: # i18n: column positioning for "hg log" _("bookmark: %s\n") % bookmark becomes tname = _(("%s:" % ns.templatename).ljust(13) + "%s\n") % name when name == 'bookmark'. The ljust(13) keeps the strings and whitespace equal. Adding a new namespace is even easier now because the log output code doesn't need to change. A future programmer would just need to add the string to the corresponding .po file (which is the same as they would have had to do previously).

tests: add a i18n translation test for log output Upcoming patches will change the way that log output is generated so we add a test to ensure that the words 'branches', 'bookmarks', and 'tags' are still translated.

filectx: fix annotate to not directly instantiate filectx 28a302e9225d changed basefilectx.annotate() to directly instantiate new filectx's instead of going through self.filectx(), this breaks extensions that replace the filectx class, and would also break future uses that would need memfilectx's.

largefiles: don't print files as both large and normal in addremove dryruns

largefiles: return the list of added files from addlargefiles() This will be used to exclude largefile candidates from the normal file matcher, which will allow add and addremove dryruns to not print a file as both a normal and a large file.

largefiles: align the output messages for an added file with core methods Core addremove prints the file relative to cwd only if patterns are provided to the command. Core add always prints relative to cwd. Also, both methods print the subrepo prefix when needed. The 'already a largefile' doesn't have an analog in core, but follows the same rules for consistency.

largefiles: align the output messages for a removed file with core methods Both cmdutil.remove() and scmutil.addremove() require verbose mode or an inexact match to print the filename. Core addremove also prints the file relative to cwd only if patterns are provided to the command. And finally, both methods print the subrepo prefix when needed.

revset: use '%' as an operator for 'only' With this patch, we can make it much easier to specify 'only(A,B)' -> A%B. Similarly, 'only(A)' -> A%. On Windows, '%' is a semi-reserved symbol in the following way: using non-bash shells (e.g. cmd.exe but NOT PowerShell, ConEmu, and cmder), %var% is only expanded when 'var' exists and is surrounded by '%'. That only leaves batch scripts which could prove to be problematic. I posit that this isn't a big issue because any developer of batch scripts already knows that to use '%' one needs to escape it by using a double '%%'. Alternatives to '%' could be '=' but that might be limiting our future if we ever decide to use temporary assignments in a revset.

transaction: support for callbacks during abort Previous transaction work added callbacks to be called during regular transaction commit/close. As part of refactoring Mozilla's pushlog extension (an extension that opens a SQLite database and tries to tie its transaction semantics to Mercurial's transaction), I discovered that the new transaction APIs were insufficient to avoid monkeypatching transaction instance internals. Adding a callback that is called during transaction abort removes the necessity for monkeypatching and completes the API.

debugnamecomplete: use new name api Instead of hardcoding a list of places to check, we use the new repo.names api to get a list of potential names to complete.

debugnamecomplete: rename from debuglabelcomplete Now that we have decided on the use of 'name' instead of 'label' we rename this function accordingly. The old method 'debuglabelcomplete' has been left as a deprecated command so that current scripts don't break.

namespaces: add __iter__ and iteritems methods Iterating over all the namespaces is a common operation, naturally, so we add those methods now. Since we are using a sorted dictionary, this method just calls the underlying __iter__ or iteritems method.

namespaces: add 'listnames' property Currently, we have no way to list all the names in a given namespace. This is needed for things such as tab completion. Future patches will use this patch for exactly that purpose.

test-bundle2-exchange: create temp script in $TESTTMP, not $TESTDIR The bundle2-pushkey-hook.sh script is currently created in $TESTTMP, and leaves an untracked file in that directory (tests/) after running.

manifest: drop withflags() method, which is now unused

context: use new manifest.diff(clean=True) support This further simplifies the status code. This simplification comes at a slight performance cost for `hg export`. Before, on mozilla-central: perfmanifest tip ! wall 0.265977 comb 0.260000 user 0.240000 sys 0.020000 (best of 38) perftags ! result: 162 ! wall 0.007172 comb 0.010000 user 0.000000 sys 0.010000 (best of 403) perfstatus ! wall 0.422302 comb 0.420000 user 0.260000 sys 0.160000 (best of 24) hgperf export tip ! wall 0.148706 comb 0.150000 user 0.150000 sys 0.000000 (best of 65) after, same repo: perfmanifest tip ! wall 0.267143 comb 0.270000 user 0.250000 sys 0.020000 (best of 37) perftags ! result: 162 ! wall 0.006943 comb 0.010000 user 0.000000 sys 0.010000 (best of 397) perfstatus ! wall 0.411198 comb 0.410000 user 0.260000 sys 0.150000 (best of 24) hgperf export tip ! wall 0.173229 comb 0.170000 user 0.170000 sys 0.000000 (best of 55) The next set of patches introduces a new manifest type implemented almost entirely in C, and more than makes up for the performance hit incurred in this change.

manifest: add optional recording of clean entries to diff This makes manifest slightly easier to use for status code.

context: use manifest.diff() to compute most of status We can do a little tiny bit better by enhancing manifest.diff to optionally include files that are in both sides. This will be done in a followup patch.

trydiff: replace dodiff=True/'binary' by binarydiff=False/True

trydiff: replace 'dodiff = False' by 'continue' The 'dodiff' variable is initialized to True and may later be set to either False or "binary". When it's set to False, we skip everything after that point, so we can simplify by instead continue-ing (the loop). We can then also drop the 'if dodiff', since it will always be true.

trydiff: make addindexmeta() unconditionally add index meta The conditional-ness is not clear from the name and there is only one caller, so it's clearer to check on the call site. Moving it also makes addindexmeta() no longer close on the 'opts' variable.

trydiff: remove unused code for git index of "combined diff" We only ever produce diffs with one pre- and one post-image, so remove the code for displaying "index" lines for combined diffs.

diff: remove dead assignment

localrepo: show headline notes in commitctx before showing filenames commitctx already showed notes with filenames but didn't provide any context. It is just as relevant to know when manifest or changelog is committed. So, in addition to filenames, also show headlines 'committing files:', 'committing manifest' and 'committing changelog'.

bundle: when verbose, show what takes up the space in the generated bundle This is kind of similar to the debugbundle command but gives summarized actual uncompressed number of bytes when creating the bundle. The numbers are as usable as the bundle format is efficient. Hopefully bundle2 will make it a better indicator of actual entropy. This is useful when accepting pull requests to assess whether the repo size increase seems reasonable for the diff before pushing stuff upstream, It has helped me catching large files that should have been committed as largefiles but was committed as regular files in intermediate changesets. This output doesn't combine well with debug output so we only enable it when verbose without debug.

setdiscovery: factorize similar sampling code We are using full sampling of 'fullsamplesize' in both case. The only difference is the debug message. So we factorise the sampling code and put the message in an extra conditional. This is going to help making changes around the sampling logic. Such changes are needed to improve discovery performance on highly headed repository.

setdiscovery: drop shadowed 'undecided' assignment The 'undecided' variable was never used before being overwritten a few lines later.

hgweb: extract changelist entry generation into own function I want to supplement changelist entries (used by shortlog and changelog endpoints) with custom metadata from an extension. i.e. I have extra per-changeset metadata that I wish to make available to templates so it can be rendered on hgweb. To facilitate this, I've extracted the logic for creating a changeset data structure into its own function, where it can be wrapped by extensions. Ideally, hgweb would use the same templater as the command line and have full access to templatekw.keywords. But that's a lot of work. This patch gets us some of the benefit without all the work. Many other hgweb commands could benefit from similar refactorings. I'm going to wait to see how this patch is received before I go crazy on extracting inline functions.

templates: use CSS classes for diff styling Use of inline style for diff styling led to significant browser memory usage on large diffs. Moving the styling into CSS classes corrects this. This patch is based on work from https://bugzilla.mozilla.org/show_bug.cgi?id=766952 and https://hg.mozilla.org/hgcustom/version-control-tools/rev/2c355a580af6

templates: remove unnecessary <span> The <span> on diffline was useless. It was only making browsers work harder. Remove it.

revset: introduce new operator "##" to concatenate strings/symbols at runtime Before this patch, there is no way to concatenate strings at runtime. For example, to search for the issue ID "1234" in descriptions against all of "issue 1234", "issue:1234", issue1234" and "bug(1234)" patterns, the revset below should be written fully from scratch for each issue ID. grep(r"\bissue[ :]?1234\b|\bbug\(1234\)") This patch introduces new infix operator "##" to concatenate strings/symbols at runtime. Operator symbol "##" comes from the same one of C pre-processor. This concatenation allows parametrizing a part of strings in revset queries. In the case of example above, the definition of the revset alias using operator "##" below can search issue ID "1234" in complicated patterns by "issue(1234)" simply: issue($1) = grep(r"\bissue[ :]?" ## $1 ## r"\b|\bbug\(" ## $1 ## r"\)") "##" operator does: - concatenate not only strings but also symbols into the string Exact distinction between strings and symbols seems not to be convenience, because it is tiresome for users (and "revset.getstring" treats both similarly) For example of revset alias "issue()", "issue(1234)" is easier than "issue('1234')". - have higher priority than any other prefix, infix and postfix operators (like as "##" of C pre-processor) This patch (re-)assigns the priority 20 to "##", and 21 to "(", because priority 19 is already assigned to "-" as prefix "negate".

largefiles: pass a matcher instead of a raw file list to removelargefiles() This is consistent with addlargefiles(), and will make it easier to get the paths that are printed correct when recursing into subrepos or invoking from outside the repository. It also now restricts the path that the addremove is performed on if a path is given, as is done with normal files. The repo.status() call needs to exclude clean files when performing an addremove, because the addremove override method calling this used to pass the list of files to delete, which caused the matcher to only consider those files in building the status list. Now the matcher is restricted only to the extent that the caller requested- usually directories if at all. There's no reason for addremove to care about clean files anyway- we don't want them deleted.

hgweb: allow viewing diffs against p1 or p2 for merge commits (issue3904) This adds UI portion of the feature that has resided in mercurial since 2012. Back then the interface was added together with the code, but was shortly backed out because it was deemed "not ready". Code, however, stayed. For the original feature and its implementation, see issue2810 and d605a82cf189. In short, the backed-out interface had two outstanding issues: 1. it was introducing an entirely new term (baseline) and 2. it was present on every changeset's page, even for changesets with 1 parent (or no parents), which didn't make sense This patch implements a hopefully better interface because: 1. it uses the usual terms (diff) and 2. it only shows up when there actually are 2 parents.

namespaces: remove templatename method on the namespaces object Now that there is an object for each individual namespace, we use the templatename property on that object instead of the method on the collection of namespaces.

namespaces: remove names method on the namespaces object Now that there is an object for each individual namespace, we use that instead of the method on the collection of namespaces.

templatekw: update namespace calls Previous patches changed the namespace api to be more of an object-oriented approach. This patch updates the template function to use said api changes.

namespaces: add __getitem__ property Since the namespaces object uses an underlying (sorted) dictionary to store the namespaces, it makes sense to expose this to naturally gain access to those namespaces.

merge with stable

cmdutil.jsonchangeset: properly compute added and removed files jsonchangeset._show() was computing the reverse status of the current changeset. As a result, added files were showing up as removed and removed files were showing up as adds. There were existing tests for this code and they were flat out wrong.

largefiles: convert addlargefiles() to vfs

rebase: clarify comment about merge ancestor when rebasing merges The code for picking a merge ancestor when rebasing merges had a long and incorrect comment. The comment would perhaps have been fine as commit message but does not make the code more readable or maintainable and is a bad substitute for correct and readable code. The correct essense of the comment is quite trivial: a merge of an ancestor of the rebase destination and an 'outside' revision can be rebased as if it was a linear change, using 'destination ancestor parent' as base and pretty much ignoring the 'outside' revision. The code path where the comment is placed is however also used for other kinds of merge rebases. The comment is thus not really correct and not helpful. I think it would be better to drop the comment and rewrite the code.

status: don't list files as both clean and deleted Tracked files that are deleted should always be reported as such, no matter what their state was in earlier revisions. This is encoded in in two conditions in the loop in basectx._buildstatus() for modified and added files, but the check is missing for clean files. We should check for clean files too, but instead of adding the check in a third place, move it earlier and skip most of the loop body for deleted files.

status: don't list files as both removed and deleted When calculating status involving the working copy and a revision other than the parent of the working copy, the files that are not in the working context manifest ('mf2' in the basectx._buildstatus()) will be reported as removed (note that deleted files _are_ in the working context manifest). However, if the file is reported as deleted in the dirstate, it will get that status too (as shown by failing tests). Fix by removing deleted files from the 'removed' list after the main loop in _buildstatus().

revset-filelog: handle hidden linkrev for file missing for head (issue4490) The fix for linkrev pointing to hidden revision was crashing when the file was missing from head's manifest. We now properly handle this case. (yes I feel silly)

run-tests: automatically add (glob) to "saved backup bundle to" lines Avoid spending too much time adding (glob) after running run-tests -i. This doesn't handle all cases but it helps. The run-tests tests add a bit of escaping of trailing (glob) in the output to avoid interference from the outer test runner. The regexp for matching the output lines contains a group for making multiline substitute in a way that works with Python before 2.7.

transaction: use the right location when cleaning up backup file (issue4479) The location variable fetch from the loop and the one used to actually fetch it mismatched. We fix the name to ensure file outside of store are cleaned up.

merge with stable

revset: delay showing parse error for the revset alias until it is referred Before this patch, a problematic revset alias aborts execution immediately, even if it isn't referred in the specified revset. If old "hg" may be used too (for example, bisecting Mercurial itself), it is also difficult to write alias definitions using features newly introduced by newer "hg" into configuration files, because such alias definitions cause unexpected abortion at parsing revset aliases with old "hg". This patch delays showing parse error for the revset alias until it is actually referred at runtime. This patch detects referring problematic aliases in "_expandaliases" by examination of "revsetalias.error", which is initialized with the error message only when parsing fails. For usability, this patch also warns about problematic aliases, even if they aren't referred at runtime. This should help users to know potential problems in their alias definitions earlier.

revset: drop pre-lazyset optimization for stringset of subset == entire repo It was introduced at e44ebd2a142a, where spanset.__contains__() did not exist. Nowadays, we have to pay huge penalty for len(subset). The following example showed that OR operation could be O(n * m^2) (n: len(repo), m: number of OR operators, m >= 2) probably because of filteredset.__len__. revset #0: 0|1|2|3|4|5|6|7|8|9 0) wall 8.092713 comb 8.090000 user 8.090000 sys 0.000000 (best of 3) 1) wall 0.445354 comb 0.450000 user 0.430000 sys 0.020000 (best of 22) 2) wall 0.000389 comb 0.000000 user 0.000000 sys 0.000000 (best of 7347) (0: 3.2.4, 1: 3.1.2, 2: this patch)

keyword: update copyright year

keyword: use vfs.reljoin and util.unlinkpath to remove kwdemo

largefiles: properly sync lfdirstate after removing largefiles The more aggressive synchronization of lfdirstate that was backed out in 1265a3a71d75 masked the problem where lfdirstate would hold an 'R' for a largefile that was added and then removed without a commit between. We could just conditionally call lfdirstate.drop() or lfdirstate.remove() here, but this also properly updates lfdirstate if the standin doesn't exist for the file somehow (i.e. call drop instead of remove). Without this change, the precommit status in the commit command immediately after the test change lists the removed (and never committed) largefile as 'R'. It can also lead to situations where the status command reports the same, long after the commit [1]. [1] http://www.selenic.com/pipermail/mercurial-devel/2015-January/065153.html

linkrev-filelog: handle filtered linkrev with no visible children (issue4307) If the file revision with a filtered linkrev does not have any (unfiltered) children, we cannot use it to bound the search for another introduction. Instead, we have to look at the file revision used by each head changeset. If one of them uses this file revision, we know there is another occurrence and we have a starting point. See inline comments for details. Adding some kind of permanent reference of all the introductions of a file revision instead of just the first one would be much better. But this is more difficult. I hope to take that into account in the next repository format.

linkrev: work around linkrev to filtered entry in 'filelog' revset This revset is used by 'hg log FILENAME'. This prevent bugs when used on a repository with hidden revisions. Instead of just discarding file revisions whose linkrevs point to filtered revisions, we put them aside and post-process them trying to find a non-filtered introduction. See inline documentation for details about how it works. This only fixes some of the problems. Once again, more will be needed when we can cannot rely on child revisions of a file to find linkrev-shadowned revisions. A test is added for 'hg log' catching such cases.

namespaces: update documentation and code indentation The previous patch changed just the functionality, while this patch focuses on the documentation and indentation to keep review simple.

namespaces: use namespace object instead of dictionary This isn't as bad as the diff seems, it only looks like scary. In this patch, we use the 'namespace' object instead of accessing keys in a dictionary. This required the 'templatename' and 'names' method to change their implementation. Later, we will remove these functions entirely due to a better api.

namespaces: copy implementation to new namespace object

namespaces: add a namespace object Currently, we use a dictionary object to store the namespace properties. This is python so let's use an object. This will allow us to be more flexible in our method signatures in the future.

localrepo: use vfs.reljoin rather than os.path.join in the localrepository class The removes the last remaining usage of os.path.join inside teh localrepository class.

largefiles: drop a redundant check for patterns when removing without --after This is in the way of passing a matcher to removelargefiles(). This method is called in exactly two places- first in overrides.addremove() (but only if the pattern list passed to it is not empty), and second in the commands.remove() override. But since the latter calls commands.remove() first, which also does this check, it isn't needed here.

context: override _dirstatestatus in workingcommitctx for correct matching Before this patch, the result of "status()" on "workingcommitctx" may incorrectly contain files other than ones to be committed, because "workingctx._dirstatestatus()" returns the result of "dirstate.status()" directly. For correct matching, this patch overrides "_dirstatestatus" in "workingcommitctx" and makes it return matched files only in "self._status". This patch uses empty list for "deleted", "unknown" and "ignored" of status, because status between "changectx"s also makes them empty.

context: avoid breaking already fixed self._status at ctx.status() Before this patch, "status()" on "workingcommitctx" with "always match" object causes breaking "self._status" in "workingctx._buildstatus()", because "workingctx._buildstatus()" caches the result of "dirstate.status()" into "self._status" for efficiency, even though it should be fixed at construction for committing. For example, template function "diff()" without any patterns in "committemplate" implies "status()" on "workingcommitctx" with "always match" object, via "basectx.diff()" and "patch.diff()". Then, broken "self._status" causes committing unexpected files. To avoid breaking already fixed "self._status" at "ctx.status()", this patch overrides "_buildstatus" in "workingcommitctx". This patch doesn't write out the result of template function "diff()" in "committemplate" in "test-commit.t", because matching against files to be committed still has an issue fixed in subsequent patch.

context: add workingcommitctx for exact context to be committed Before this patch, "workingctx" is also used for the context to be committed. But "workingctx" works incorrectly in some cases. For example, even when only some of changed files in the working directory are committed, "status()" on "workingctx" object for committing recognizes files not to be committed as changed, too. As the preparation for fixing these issues, this patch chooses adding new class "workingcommitctx" for exact context to be committed, because switching by the flag (like "self._fixedstatus" or so) in some code paths of "workingctx" is less readable and maintenancable.

context: make unknown/ignored/clean of cached status empty for equivalence Before this patch, "workingctx.status" caches the result of "dirstate.status" directly into "self._status". But "dirstate.status" is invoked with False "list*" arguments in normal "self._status" accessing route, and this makes "unknown"/"ignored"/"clean" of status empty. This may cause unexpected result of code paths internally accessing to them (accessors for external usage are already removed by previous patch). This patch makes "unknown"/"ignored"/"clean" of cached status empty for equivalence. Making them empty is executed only when at least one of "unknown", "ignored" or "clean" has files, for efficiency.

templatefilters.json: stabilize output The json filter was previously iterating over keys in an object in an undefined order. Let's throw a sorted() in there so output is consistent. It's somewhat frightening that there are no tests for the json filter. Subsequent commits will add them, so we pass on the opportunity to add them here.

templatefilters.json: call functions The "changeset" template from hgweb is using a lambda in the "diffsummary" key. In preparation for enabling JSON output from hgweb, teach the json filter how to call functions.

merge with stable

linkrev: also adjust linkrev when bootstrapping annotate (issue4305) The annotate logic now use the new 'introrev' method to bootstrap its traversal. This catches issues from linkrev-shadowing of the changeset introducing the version of a file in source changeset. More tests have been added to display pathological cases.

linkrev: also adjust linkrev when bootstrapping 'follow' revset The follow revset (used by `hg log --follow`) now uses the new 'introrev' method to bootstrap its traversal. This catches issues from linkrev-shadowing of the changesets introducing the version of a file in source changeset. A new test has been added to display pathological cases. Another test is affected because it was meant to test this behavior but actually failed to do so for half of the output. The output are now similar.

linkrev: introduce an 'introrev' method on filectx The previous changeset properly fixed the ancestors computation, but we need to ensure that the initial filectx is also using the right changeset. When asking for log or annotation from a certain point, the first step is to define the changeset that introduced the current file version. We cannot just pick the "starting point" changesets as it may just "use" the file revision, unchanged. Currently, we were using 'linkrev' for this purpose, but this exposes us to unexpected branch-jumping when the revision introducing the starting point version is itself linkrev-shadowed. So we need to take the topology into account again. Therefore, we introduce an 'introrev' function, returning the changeset which introduced the file change in the current changeset. This function will be used to fix linkrev-related issues when bootstrapping 'hg log --follow' and 'hg annotate'. It reuses the '_adjustlinkrev' function, extending it to allow introspection of the initial changeset too. In the previous usage of the '_adjustlinkrev' the starting rev was always using a children file revisions, so it could be safely ignored in the search. In this case, the starting point is using the revision of the file we are looking, and may be the changeset we are looking for.

filectx.parents: enforce changeid of parent to be in own changectx ancestors Because of the way filenodes are computed, you can have multiple changesets "introducing" the same file revision. For example, in the changeset graph below, changeset 2 and 3 both change a file -to- and -from- the same content. o 3: content = new | | o 2: content = new |/ o 1: content = old In such cases, the file revision is create once, when 2 is added, and just reused for 3. So the file change in '3' (from "old" to "new)" has no linkrev pointing to it). We'll call this situation "linkrev-shadowing". As the linkrev is used for optimization purposes when walking a file history, the linkrev-shadowing results in an unexpected jump to another branch during such a walk.. This leads to multiple bugs with log, annotate and rename detection. One element to fix such bugs is to ensure that walking the file history sticks on the same topology as the changeset's history. For this purpose, we extend the logic in 'basefilectx.parents' so that it always defines the proper changeset to associate the parent file revision with. This "proper" changeset has to be an ancestor of the changeset associated with the child file revision. This logic is performed in the '_adjustlinkrev' function. This function is given the starting changeset and all the information regarding the parent file revision. If the linkrev for the file revision is an ancestor of the starting changeset, the linkrev is valid and will be used. If it is not, we detected a topological jump caused by linkrev shadowing, we are going to walk the ancestors of the starting changeset until we find one setting the file to the revision we are trying to create. The performance impact appears acceptable: - We are walking the changelog once for each filelog traversal (as there should be no overlap between searches), - changelog traversal itself is fairly cheap, compared to what is likely going to be perform on the result on the filelog traversal, - We only touch the manifest for ancestors touching the file, And such changesets are likely to be the one introducing the file. (except in pathological cases involving merge), - We use manifest diff instead of full manifest unpacking to check manifest content, so it does not involve applying multiple diffs in most case. - linkrev shadowing is not the common case. Tests for fixed issues in log, annotate and rename detection have been added. But this changeset does not solve all problems. It fixes -ancestry- computation, but if the linkrev-shadowed changesets is the starting one, we'll still get things wrong. We'll have to fix the bootstrapping of such operations in a later changeset. Also, the usage of `hg log FILE` without --follow still has issues with linkrev pointing to hidden changesets, because it relies on the `filelog` revset which implement its own traversal logic that is still to be fixed. Thanks goes to: - Matt Mackall: for nudging me in the right direction - Julien Cristau and Rémi Cardona: for keep telling me linkrev bug were an evolution show stopper for 3 years. - Durham Goode: for finding a new linkrev issue every few weeks - Mads Kiilerich: for that last rename bug who raise this topic over my anoyance limit.

context: remove unreliable accessor methods from committablectx There are two caching routes for (propertycache-ed) "_status" below in committablectx: - invoking "status()": "dirstate.status()" is invoked, and the result of it is cached into "_status". In this case, any of "listignored", "listclean" and "listunknown" may be True. - accessing "_status" directly before "status()": Own "status()" is invoked, but all of "listignored", "listclean" and "listunknown" arguments are False, in this case. "ignored"/"clean"/"unknown" accessor methods of "committablectx" use corresponded fields of "_status", but these fields aren't reliable, because these fields are empty when: - "_status" method is executed before accessors, or - "status()" is executed with "list*=False" before accessors In addition to it, these accessors aren't used in the recent Mercurial implementation. At least, removing them doesn't cause any test failures.

context: cache self._status correctly at workingctx.status Before this patch, "workingctx.status" always replaces "self._status" by the recent result, even though: - status isn't calculated against the parent of the working directory, or - specified "match" isn't "always" one (status is only visible partially) If "workingctx" object is shared between some procedures indirectly referring "ctx._status", this incorrect caching may cause unexpected result: for example, "ctx._status" is used via "manifest()", "files()" and so on. To cache "self._status" correctly at "workingctx.status", this patch overwrites "self._status" in "workingctx._buildstatus" only when: - status is calculated against the parent of the working directory, and - specified "match" is "always" one This patch can be applied (and effective) only on default branch, because procedure around "basectx.status" is much different between stable and default: for example, overwriting "self._status" itself is executed not in "workingctx._buildstatus" but in "workingctx._poststatus", on stable branch.

filectx.parents: also fetch the filelog of rename source too we are going to need this filelog for the linkrev adjustment, so we better normalise the list and have the filelog in all case. This is done in a previous changeset to help readability.

Added signature for changeset 1265a3a71d75

Added tag 3.2.4 for changeset 1265a3a71d75

largefiles: backout f72d73937853 - linear updates handle m -> a differently f72d73937853 introduced a significant performance regression: All largefiles were marked 'normallookup' by linear (or noop) updates and had to be rehashed by the next command. The previous change introduced a different solution to the problem f72d73937853 solved and we can thus back it out again.

largefiles: mark lfile as added in lfdirstate when the standin is added This is an alternative solution to the problem addressed by f72d73937853. This implementation has the advantage that it doesn't mark clean largefiles as normallookup. We can thus avoid repeated rehashing of all largefiles when f72d73937853 is backed out. This implementation use the existing 'lfmr' actions that 23fe278bde43 introduced for handling another part of the same cases.

tests: add test coverage for lfdirstate invalidation of linear update f72d73937853 introduced a significant performance regression: All largefiles are marked 'normallookup' in lfdirstate by linear (or noop) updates and has to be rehashed by the next command. To avoid such regressions, keep an eye on the dirstate content after a plain 'hg up'.