merge with stable

unpacker: check the right exception type for 2.4

hgweb: fix diffstat links in paper/changeset.tmpl '<a .../>foo</a>' syntax is incorrect, since the first tag just "tries" to close itself and then the actual content follows. It doesn't work, either because web browsers know better than this or because there should be a whitespace before /: '<a />'. So for the hgweb users the links looked normal anyway, but now they are correct in code as well.

hgweb: close <img> elements Templates declare xhtml doctype, which means, in particular, that the document must also be valid xml. So <img> elements must be closed.

hgweb: close <p> elements <p> elements can only contain inline elements, so as soon as browser encounters a block element (e.g. block <div>) "inside" a <p>, it puts an implicit </p>. It's better to do this explicitly.

hgweb: close <th> properly in spartan/filelogentry.tmpl

revset: simplify fullreposet.__and__ to call sort() with boolean flag Note that sort() takes a boolean flag, so other.sort(reverse) was wrong. It just worked fine because there is a top-level function, reverse().

revset: fix spanset.isascending() to honor sort() or reverse() request Because spanset.isascending() ignored the ascending flag, the result of "fullreposet() & x" was always sorted in ascending order. The test case is carefully chosen to call fullreposet.__and__.

hghave: we now support Python 2.7.9's ssl for https

Makefile.python: try curl if wget fails Macs ship with curl and not wget, so this is a nice little tweak for folks testing on OS X.

test-https: glob error messages more so we pass on Python 2.7.9 Python 2.7.9 cleans up how it stringifies SSL errors, so we have to look only for the important bit (certificate verify failed) rather than looking for specific ssl module goop (which is now unstable).

filelog: fix backwards comment for 'backrevref'

filelog: remove trailing "form feed" character

filelog: remove unused variable 'lkr' It's used further down, but it's overwritten before, so it's technically a dead assignment, but unnecessary nevertheless.

branchmap: add seek() to end of file before calling tell() on append open() This is similar to 48c232873a54, which was subsequently modified in 19f5dec2d61f for 2.4. Unexpected test changes on Windows occurred without this.

tests: fix test-casefolding.t output for branchcache This belongs with 7d63398fbfd1. I assume that the failure to read is OK, because there is similar output in test-convert-svn-encoding.t.

setdiscovery: remove '_setupsample' function It is now unused.

setdiscovery: document '_takequicksample'

setdiscovery: drop '_setupsample' usage in '_takequicksample' For '_takefullsample' we can just retrieve the list of head directly and ignore the rest of the complex return values. This was the last call to the infamous '_updatesample' function.

setdiscovery: drop the 'always' argument to '_updatesample' This argument exists because of the complex code flow in '_takequicksample'. It first gets the list of heads and then calls '_updatesample' on an empty initial sample and a size limit matching the differences between the number of heads and the target sample size. Finally the heads and the sample from '_updatesample' were added. To ensure this addition result had the exact target length, the code had to ensure no elements from the heads were added to the '_updatesample' content and therefore was passing this "always included set of heads". Instead we can just update the initial heads sample directly and use the final target size as target size for the update. This removes the need for this 'always' parameter to the '_updatesample' function The test are affected because different set building order results in different random sampling.

setdiscovery: always add exponential sample to the heads As explained in a previous changeset, prioritizing heads too much behaves pathologically when there are more heads than the sample size. To counter this, we always inject exponential samples before reducing to the sample size limit. This already show some benefit in the test themselves, but on a real-world example this moves my discovery for push to pathologically headed repo from 45 rounds to 17 of them. We should maybe ensure that at least 25% of the result sample is heads, but I think the random sampling will be fine in practice.

setdiscovery: directly run '_updatesample' The heads and exponential sample are going to end up in the same set before any extra processing happens. We simplify the code by directly updating a set with heads. Changes in the order the set is built lead to small changes in the random sampling output. But after double checking, I can confirm the input data to the random sampling is consistent.

setdiscovery: stop using '_setupsample' in '_takefullsample' Very few of the return values of '_setupsample' remain in use, so we directly retrieve the value we care about and drop the '_setupsample' call.

setdiscovery: randomly pick between heads and sample when taking full sample Before this changeset, the discovery protocol was too heads-centric. Heads of the undiscovered set were always sent for discovery and any room remaining in the sample were filled with exponential samples (and random ones if any room remained). This behaved extremely poorly when the number of heads exceeded the sample size, because we keep just asking about the existence of heads, then their direct parent and so on. As a result, the 'O(log(len(repo)))' discovery turns into a 'O(len(repo))' one. As a solution we take a random sample of the heads plus exponential samples. This way we ensure some exponential sampling is achieved, bringing back some logarithmic convergence of the discovery again. This patch only applies this principle in one place. More places will be updated in future patches. One test is impacted because the random sample happen to be different. By chance, it helps a bit in this case.

setdiscovery: document the '_updatesample' function This function is central in the sample building process, having it documented help code readability a lot.

setdiscovery: avoid calling any sample building if the undecided set is small If the length of undecided is smaller than the sample size, we can just request information for all of them. This conditional was previously handled by '_setupsample'. But '_setupsample' is in my opinion a problematic function with blurry semantics. Having this conditional explicitly earlier makes the code more explicit and moves us closer to removing this '_setupsample' function.

setdiscovery: delay sample building calls to gather them in a single place Some of the logic around sample building is duplicated in the sample builders, it would clean up thing to extract it in the top function, but this requires all codes to be in the same place. This changeset mostly exists to make the next one more clear.

setdiscovery: drop unused 'initial' argument for '_takequicksample' There is a single call site, and it is always using 'initial=True'. So we just drop the argument and the associated condition.

readmarkers: add a SHA256 fixme note

readmarkers: fast-path single successors and parents This gives about a 5% performance bump.

readmarkers: promote global constants to locals for performance

readmarkers: drop a temporary

readmarkers: read node reading into node length conditional This removes some conditional assignments

readmarkers: drop a temporary Two other temporaries are renamed to fit line-length.

readmarkers: hoist subtraction out of loop comparison

readmarkers: streamline offset tracking This minimizes the number of assignments and operations needed to use offsets.

readmarkers: use unpacker for fixed header

readmarkers: drop metadata temporary

readmarkers: drop date temporary

readmarkers: drop another conditional

readmarkers: drop a conditional

readmarkers: add some whitespace

readmarkers: combine parent conditionals

readmarkers: drop temporary substring assignments Assignments are expensive in inner loops

util: introduce unpacker This allows taking advantage of Python 2.5+'s struct.Struct, which provides a slightly faster unpack due to reusing formats. Sadly, .unpack_from is significantly slower.

perf: add a configurable sleep on startup This is intended to counteract power management by giving a consistent idle period before test runs.

revset: use localrepo revbranchcache for branch name filtering Branch name filtering in revsets was expensive. For every rev it created a changectx and called .branch() which retrieved the branch name from the changelog. Instead, use the revbranchcache. The revbranchcache is used read-only. The revset implementation with generators and callbacks makes it hard to figure out when we are done using/updating the cache and could write it back. It would also be 'tricky' to lock the repo for writing from within a revset execution. Finally, the branchmap update will usually make sure that the cache is updated before any revset can be run. The revbranchcache is used without any locking but is short-lived and used in a tight loop where we can assume that the changelog doesn't change ... or where it not is relevant to us if it does. perfrevset 'branch(mobile)' on mozilla-central. Before: ! wall 10.989637 comb 10.970000 user 10.940000 sys 0.030000 (best of 3) After, no cache: ! wall 7.368656 comb 7.370000 user 7.360000 sys 0.010000 (best of 3) After, with cache: ! wall 0.528098 comb 0.530000 user 0.530000 sys 0.000000 (best of 18) The performance improvement even without cache come from being based on branchinfo on the changelog instead of using ctx.branch(). Some tests are added to verify that the revbranchcache works and keep an eye on when the cache files actually are updated.

branchmap: use revbranchcache when updating branch map The revbranchcache is read on demand before it will be used for updating the branch map. It is written back when the branchmap is written and it will thus use the same locking as branchmap. The revbranchcache instance is short-lived; it is only stored in the branchmap from .update() is invoked and until .write() is invoked. Branchmap already assume that the repo is locked in that case. The use of revbranchcache for branch map updates will make sure that the revbranchcache "always" is kept up-to-date. The perfbranchmap benchmark is somewhat bogus, especially when we can see that the caching makes a significant difference between the realistic case of a first run and the rare case of rerunning it with a full cache. Here are some 'base' numbers on mozilla-central: Before: ! wall 6.912745 comb 6.910000 user 6.840000 sys 0.070000 (best of 3) After - initial, cache is empty: ! wall 7.792569 comb 7.790000 user 7.720000 sys 0.070000 (best of 3) After - cache is full: ! wall 0.879688 comb 0.880000 user 0.870000 sys 0.010000 (best of 4) The overhead when running with empty cache comes from checking, missing and updating it every time. Most of the performance improvement comes from not having to extract the branch info from the changelog. The last doubling of performance comes from no longer having to convert all branch names to local encoding but reuse the few already converted branch names. On the hg repo: Before: ! wall 0.715703 comb 0.710000 user 0.710000 sys 0.000000 (best of 14) After: ! wall 0.105489 comb 0.110000 user 0.110000 sys 0.000000 (best of 87)

branchcache: introduce revbranchcache for caching of revision branch names It is expensive to retrieve the branch name of a revision. Very expensive when creating a changectx and calling .branch() every time - slightly less when using changelog.branchinfo(). Now, to speed things up, provide a way to cache the results on disk in an efficient format. Each branchname is assigned a number, and for each revision we store the number of the corresponding branch name. The branch names are stored in a dedicated file which is strictly append only. Branch names are usually reused across several revisions, and the total list of branch names will thus be so small that it is feasible to read the whole set of names before using the cache. It will however do that it might be more efficient to use the changelog for retrieving the branch info for a single revision. The revision entries are stored in another file. This file is usually append only, but if the repository has been modified, the file will be truncated and the relevant parts rewritten on demand. The entries for each revision are 8 bytes each, and the whole revision file will thus be 1/8 of 00changelog.i. Each revision entry contains the first 4 bytes of the corresponding node hash. This is used as a check sum that always is verified before the entry is used. That check is relatively expensive but it makes sure history modification is detected and handled correctly. It will also detect and handle most revision file corruptions. This is just a cache. A new format can always be introduced if other requirements or ideas make that seem like a good idea. Rebuilding the cache is not really more expensive than it was to run for example 'hg log -b branchname' before this cache was introduced. This new method is still unused but promise to make some operations several times faster once it actually is used. Abandoning Python 2.4 would make it possible to implement this more efficiently by using struct classes and pack_into. The Python code could probably also be micro optimized or it could be implemented very efficiently in C where it would be easy to control the data access.

hgweb: move archive entries outside of <li> in monoblue style archiveentry already includes surrounding <li></li>, so putting archive entries inside <li> element produced incorrect markup.

hgweb: add searchhint to templates/coal/map coal style uses every template (except header.tmpl) directly from paper style, but doesn't use paper/map file. Elements defined in such map files are used in templates as you would expect. For example, paper/search.tmpl contains '{searchhint}' and template engine replaces that with the actual hint. But when coal style reuses paper/search.tmpl, it needs to define searchhint in its map file as well, or template engine will not find it. So let's copy it from paper/map to coal/map. Before this change, if the coal style was selected, the hint for the search field in page header was present, but it was completely empty. Although the absence of searchhint in coal/map produced no error.

largefiles: enable subrepo support for remove Previously, remove failed when operating on a largefile in a subrepo, stating that the file is untracked.

transplant: properly skip empty changeset (issue4423) If resolving a merge conflict result in an empty changesets, we now properly skip the changeset instead of crashing. Original patch from Robert Collins <robertc@robertcollins.net>.

color: add support for colorizing git subrepo diffs

tests: fix test-casefolding.t output This change started with a387b0390082.

synthrepo: new filenames must not also be new directories, and vice-versa When generating many new files into a set of many possible new directories, there is the possibility that the same path is chosen as both file and directory. How likely this is depends on the size of the dictionary used, the generated directory structure and the number of generated files.

status: don't override _buildstatus() in workingcommitctx Now that the caching into _status is done in workingctx._dirstatestatus(), which workingcommitctx._dirstatestatus() does not call, there is no caching to prevent in _buildstatus(), so stop overriding it.

status: cache dirstate status in _dirstatestatus() Since it's only the dirstate status we cache, it makes more sense to cache it in the _dirstatestatus() method. Note that this change means the dirstate status will also be cached when status is requested between the working copy and some other revision, while we currently only cache the result if exactly the status between the working copy and its parent is requested.

localrepo: add ignoremissing parameter to branchtip Previously, in the namespaces api, the only caller of branchtip was singlenode which happened to raise the same exception that branchtip raised: KeyError. This is a minor change but will allow upcoming patches to use repo.branchtip to not raise an exception if a branch doesn't exist. After that, it will be possible for extensions to use the namespace api in a stable way.

namespaces: add method to return a list of nodes for a given name This is a helpful method that some extensions can make use of (e.g. for custom revsets); currently not used in core.

namespaces: add test for log Now that we have enough features in the namespaces api, we add a test for it.

log: use new namespaces api to display names The only caveat here is that branches must be displayed first due to backwards compatibility. The order of namespaces is defined to be the 'update' order which, unfortunately, is not the same as log output order. It's worth mentioning that the log output is still translated the same as before since we are formating our strings the same way: # i18n: column positioning for "hg log" _("bookmark: %s\n") % bookmark becomes tname = _(("%s:" % ns.templatename).ljust(13) + "%s\n") % name when name == 'bookmark'. The ljust(13) keeps the strings and whitespace equal. Adding a new namespace is even easier now because the log output code doesn't need to change. A future programmer would just need to add the string to the corresponding .po file (which is the same as they would have had to do previously).

tests: add a i18n translation test for log output Upcoming patches will change the way that log output is generated so we add a test to ensure that the words 'branches', 'bookmarks', and 'tags' are still translated.

filectx: fix annotate to not directly instantiate filectx 28a302e9225d changed basefilectx.annotate() to directly instantiate new filectx's instead of going through self.filectx(), this breaks extensions that replace the filectx class, and would also break future uses that would need memfilectx's.

largefiles: don't print files as both large and normal in addremove dryruns

largefiles: return the list of added files from addlargefiles() This will be used to exclude largefile candidates from the normal file matcher, which will allow add and addremove dryruns to not print a file as both a normal and a large file.

largefiles: align the output messages for an added file with core methods Core addremove prints the file relative to cwd only if patterns are provided to the command. Core add always prints relative to cwd. Also, both methods print the subrepo prefix when needed. The 'already a largefile' doesn't have an analog in core, but follows the same rules for consistency.

largefiles: align the output messages for a removed file with core methods Both cmdutil.remove() and scmutil.addremove() require verbose mode or an inexact match to print the filename. Core addremove also prints the file relative to cwd only if patterns are provided to the command. And finally, both methods print the subrepo prefix when needed.

revset: use '%' as an operator for 'only' With this patch, we can make it much easier to specify 'only(A,B)' -> A%B. Similarly, 'only(A)' -> A%. On Windows, '%' is a semi-reserved symbol in the following way: using non-bash shells (e.g. cmd.exe but NOT PowerShell, ConEmu, and cmder), %var% is only expanded when 'var' exists and is surrounded by '%'. That only leaves batch scripts which could prove to be problematic. I posit that this isn't a big issue because any developer of batch scripts already knows that to use '%' one needs to escape it by using a double '%%'. Alternatives to '%' could be '=' but that might be limiting our future if we ever decide to use temporary assignments in a revset.

transaction: support for callbacks during abort Previous transaction work added callbacks to be called during regular transaction commit/close. As part of refactoring Mozilla's pushlog extension (an extension that opens a SQLite database and tries to tie its transaction semantics to Mercurial's transaction), I discovered that the new transaction APIs were insufficient to avoid monkeypatching transaction instance internals. Adding a callback that is called during transaction abort removes the necessity for monkeypatching and completes the API.

debugnamecomplete: use new name api Instead of hardcoding a list of places to check, we use the new repo.names api to get a list of potential names to complete.

debugnamecomplete: rename from debuglabelcomplete Now that we have decided on the use of 'name' instead of 'label' we rename this function accordingly. The old method 'debuglabelcomplete' has been left as a deprecated command so that current scripts don't break.

namespaces: add __iter__ and iteritems methods Iterating over all the namespaces is a common operation, naturally, so we add those methods now. Since we are using a sorted dictionary, this method just calls the underlying __iter__ or iteritems method.

namespaces: add 'listnames' property Currently, we have no way to list all the names in a given namespace. This is needed for things such as tab completion. Future patches will use this patch for exactly that purpose.

test-bundle2-exchange: create temp script in $TESTTMP, not $TESTDIR The bundle2-pushkey-hook.sh script is currently created in $TESTTMP, and leaves an untracked file in that directory (tests/) after running.

manifest: drop withflags() method, which is now unused

context: use new manifest.diff(clean=True) support This further simplifies the status code. This simplification comes at a slight performance cost for `hg export`. Before, on mozilla-central: perfmanifest tip ! wall 0.265977 comb 0.260000 user 0.240000 sys 0.020000 (best of 38) perftags ! result: 162 ! wall 0.007172 comb 0.010000 user 0.000000 sys 0.010000 (best of 403) perfstatus ! wall 0.422302 comb 0.420000 user 0.260000 sys 0.160000 (best of 24) hgperf export tip ! wall 0.148706 comb 0.150000 user 0.150000 sys 0.000000 (best of 65) after, same repo: perfmanifest tip ! wall 0.267143 comb 0.270000 user 0.250000 sys 0.020000 (best of 37) perftags ! result: 162 ! wall 0.006943 comb 0.010000 user 0.000000 sys 0.010000 (best of 397) perfstatus ! wall 0.411198 comb 0.410000 user 0.260000 sys 0.150000 (best of 24) hgperf export tip ! wall 0.173229 comb 0.170000 user 0.170000 sys 0.000000 (best of 55) The next set of patches introduces a new manifest type implemented almost entirely in C, and more than makes up for the performance hit incurred in this change.

manifest: add optional recording of clean entries to diff This makes manifest slightly easier to use for status code.

context: use manifest.diff() to compute most of status We can do a little tiny bit better by enhancing manifest.diff to optionally include files that are in both sides. This will be done in a followup patch.

trydiff: replace dodiff=True/'binary' by binarydiff=False/True

trydiff: replace 'dodiff = False' by 'continue' The 'dodiff' variable is initialized to True and may later be set to either False or "binary". When it's set to False, we skip everything after that point, so we can simplify by instead continue-ing (the loop). We can then also drop the 'if dodiff', since it will always be true.

trydiff: make addindexmeta() unconditionally add index meta The conditional-ness is not clear from the name and there is only one caller, so it's clearer to check on the call site. Moving it also makes addindexmeta() no longer close on the 'opts' variable.

trydiff: remove unused code for git index of "combined diff" We only ever produce diffs with one pre- and one post-image, so remove the code for displaying "index" lines for combined diffs.

diff: remove dead assignment

localrepo: show headline notes in commitctx before showing filenames commitctx already showed notes with filenames but didn't provide any context. It is just as relevant to know when manifest or changelog is committed. So, in addition to filenames, also show headlines 'committing files:', 'committing manifest' and 'committing changelog'.

bundle: when verbose, show what takes up the space in the generated bundle This is kind of similar to the debugbundle command but gives summarized actual uncompressed number of bytes when creating the bundle. The numbers are as usable as the bundle format is efficient. Hopefully bundle2 will make it a better indicator of actual entropy. This is useful when accepting pull requests to assess whether the repo size increase seems reasonable for the diff before pushing stuff upstream, It has helped me catching large files that should have been committed as largefiles but was committed as regular files in intermediate changesets. This output doesn't combine well with debug output so we only enable it when verbose without debug.

setdiscovery: factorize similar sampling code We are using full sampling of 'fullsamplesize' in both case. The only difference is the debug message. So we factorise the sampling code and put the message in an extra conditional. This is going to help making changes around the sampling logic. Such changes are needed to improve discovery performance on highly headed repository.

setdiscovery: drop shadowed 'undecided' assignment The 'undecided' variable was never used before being overwritten a few lines later.

hgweb: extract changelist entry generation into own function I want to supplement changelist entries (used by shortlog and changelog endpoints) with custom metadata from an extension. i.e. I have extra per-changeset metadata that I wish to make available to templates so it can be rendered on hgweb. To facilitate this, I've extracted the logic for creating a changeset data structure into its own function, where it can be wrapped by extensions. Ideally, hgweb would use the same templater as the command line and have full access to templatekw.keywords. But that's a lot of work. This patch gets us some of the benefit without all the work. Many other hgweb commands could benefit from similar refactorings. I'm going to wait to see how this patch is received before I go crazy on extracting inline functions.

templates: use CSS classes for diff styling Use of inline style for diff styling led to significant browser memory usage on large diffs. Moving the styling into CSS classes corrects this. This patch is based on work from https://bugzilla.mozilla.org/show_bug.cgi?id=766952 and https://hg.mozilla.org/hgcustom/version-control-tools/rev/2c355a580af6

templates: remove unnecessary <span> The <span> on diffline was useless. It was only making browsers work harder. Remove it.

revset: introduce new operator "##" to concatenate strings/symbols at runtime Before this patch, there is no way to concatenate strings at runtime. For example, to search for the issue ID "1234" in descriptions against all of "issue 1234", "issue:1234", issue1234" and "bug(1234)" patterns, the revset below should be written fully from scratch for each issue ID. grep(r"\bissue[ :]?1234\b|\bbug\(1234\)") This patch introduces new infix operator "##" to concatenate strings/symbols at runtime. Operator symbol "##" comes from the same one of C pre-processor. This concatenation allows parametrizing a part of strings in revset queries. In the case of example above, the definition of the revset alias using operator "##" below can search issue ID "1234" in complicated patterns by "issue(1234)" simply: issue($1) = grep(r"\bissue[ :]?" ## $1 ## r"\b|\bbug\(" ## $1 ## r"\)") "##" operator does: - concatenate not only strings but also symbols into the string Exact distinction between strings and symbols seems not to be convenience, because it is tiresome for users (and "revset.getstring" treats both similarly) For example of revset alias "issue()", "issue(1234)" is easier than "issue('1234')". - have higher priority than any other prefix, infix and postfix operators (like as "##" of C pre-processor) This patch (re-)assigns the priority 20 to "##", and 21 to "(", because priority 19 is already assigned to "-" as prefix "negate".

largefiles: pass a matcher instead of a raw file list to removelargefiles() This is consistent with addlargefiles(), and will make it easier to get the paths that are printed correct when recursing into subrepos or invoking from outside the repository. It also now restricts the path that the addremove is performed on if a path is given, as is done with normal files. The repo.status() call needs to exclude clean files when performing an addremove, because the addremove override method calling this used to pass the list of files to delete, which caused the matcher to only consider those files in building the status list. Now the matcher is restricted only to the extent that the caller requested- usually directories if at all. There's no reason for addremove to care about clean files anyway- we don't want them deleted.

hgweb: allow viewing diffs against p1 or p2 for merge commits (issue3904) This adds UI portion of the feature that has resided in mercurial since 2012. Back then the interface was added together with the code, but was shortly backed out because it was deemed "not ready". Code, however, stayed. For the original feature and its implementation, see issue2810 and d605a82cf189. In short, the backed-out interface had two outstanding issues: 1. it was introducing an entirely new term (baseline) and 2. it was present on every changeset's page, even for changesets with 1 parent (or no parents), which didn't make sense This patch implements a hopefully better interface because: 1. it uses the usual terms (diff) and 2. it only shows up when there actually are 2 parents.

namespaces: remove templatename method on the namespaces object Now that there is an object for each individual namespace, we use the templatename property on that object instead of the method on the collection of namespaces.

namespaces: remove names method on the namespaces object Now that there is an object for each individual namespace, we use that instead of the method on the collection of namespaces.