templater: fix handling of \-escapes in raw string literals The backslash character should start escape sequences no matter if a string is prefixed with 'r'. They are just not interpreted as escape sequences in raw strings. revset.tokenize() handles them correctly, but templater didn't. https://docs.python.org/2/reference/lexical_analysis.html#string-literals

templater: evaluate "query" argument passed to revset() revset() had the same issue as 9452112c8eb0. It crashed by passing non-string expression.

match: let 'path:.' and 'path:' match everything (issue4687) Previously, both queries exited with code 1, printing nothing. The pattern in the latter query is normalized to '.', so it is really the same case.

changelog: document the 'readpending' method I happen to have spent some time understanding this logic, so I'm leaving documentation for the next poor fellow.

revsets: use '&' instead of '.filter' in head More high level operations are more likely to be optimised.

fileset: don't suggest private or undocumented queries Unlike revsets, it looks like all of the filesets are documented, so there's really nothing to test. This is aimed more at parity with revsets and future-proofing.

revset: don't suggest private or undocumented queries I noticed when I mistyped 'matching', that it suggested '_matchfiles' as well. Rather than simply exclude names that start with '_', this excludes anything without a docstring. That way, if it isn't in the help text, it isn't suggested, such as 'wdir()'.

patch: add fuzz config flag (issue4697)

devel-warn: issue a warning for old style revsets We have move to smartset class more than a year ago, we now have the tool to aggressively nudge developer into upgrading their extensions.

devel-warn: move the develwarn function as a method of the ui object We are going to use this feature in more and more place. Having to import scmutil makes it an import cycle hell.

templates: fix missing closing brace in map-cmdline.status Currently the parser does not validate the last closing brace, but future patches will make it less permissive for this kind of syntax errors.

verify: clarify misleading fncache message This is a message about cache corruption, not repository corruption or actually missing files. Fix message and reduce to a warning.

phase: add a pointer to 'hg help phases' in the 'phase' help The command and the general topic have very similar name. The topic is pointing to the command, but the command was not pointing to the topic.

publishing: unconditionally trust publishing flag Unfortunately, there are currently no tests for this behavior because we lack a good way of testing trust.

publishing: use new helper method

publishing: add helper method to localrepo

revset: make use of natively-computed set for 'draft()' and 'secret()' If the computation of a set for each phase (done in C) is available, we use it directly instead of applying a simple filter. This give a massive speed-up in the vast majority of cases. On my mercurial repo with about 15000 out of 40000 draft changesets: revset: draft() plain min first last 0) 0.011201 0.019950 0.009844 0.000074 1) 0.000284 2% 0.000312 1% 0.000314 3% 0.000315 x4.3 Bad performance for "last" come from the handling of the 15000 elements set (memory allocation, filtering hidden changesets (99% of it) etc. compared to applying the filter only on a handfuld of revisions (the first draft changesets being close of tip). This is not seen as an issue since: * Timing is still pretty good and in line with all the other one, * Current user of Vanilla Mercurial will not have 1/3 of their repo draft, This bad effect disappears when phase's set is smaller. (about 200 secrets): revset: secret() plain min first last 0) 0.011181 0.022228 0.010851 0.000452 1) 0.000058 0% 0.000084 0% 0.000087 0% 0.000087 19%

revset: refactor the non-public phase code Code for draft and secret are the same. We'll make it more complex to take advantages of the set recomputed in C, so we first refactor the code to only have one place to update (and make sure all behave properly). We do not refactor the 'public()' code because it does not have a natively computed set.

revset: translate node directly with changelog in 'head' Using 'repo[X]' is much slower because it creates a 'changectx' object and goes though multiple layers of code to do so. It is also error prone if there is tags, bookmarks, branch or other names that could map to a node hash and take precedence (user are wicked). This provides a significant performance boost on repository with a lot of heads. Benchmark result for a repo with 1181 heads. revset: head() plain min last reverse 0) 0.014853 0.014371 0.014350 0.015161 1) 0.001402 9% 0.000975 6% 0.000874 6% 0.001415 9% revset: head() - public() plain min last reverse 0) 0.015121 0.014420 0.014560 0.015028 1) 0.001674 11% 0.001109 7% 0.000980 6% 0.001693 11% revset: draft() and head() plain min last reverse 0) 0.015976 0.014490 0.014214 0.015892 1) 0.002335 14% 0.001018 7% 0.000887 6% 0.002340 14% The speed up is visible even when other more costly revset are in use revset: head() and author("mpm") plain min last reverse 0) 0.105419 0.090046 0.017169 0.108180 1) 0.090721 86% 0.077602 86% 0.003556 20% 0.093324 86%

revset: use a baseset in _notpublic() The '_notpublic()' internal revset was "returning" a set. That was wrong. We now return a 'baseset' as appropriate. This has no effect on performance in most case, because we do the exact same operation than what the combination with a 'fullreposet' was doing. This as a small effect on some operation when combined with other set, because we now apply the filtering in all cases. I think the correctness is worth the impact on some corner cases. The optimizer should take care of these corner cases anyway. revset #0: not public() plain min max first last reverse 0) 0.000465 0.000491 0.000495 0.000500 0.000494 0.000479 1) 0.000484 0.000503 0.000498 0.000505 0.000504 0.000491 revset #1: (tip~1000::) - public() plain min max first last reverse 0) 0.002765 0.001742 0.002767 0.001730 0.002761 0.002782 1) 0.002847 0.001777 0.002776 0.001741 0.002764 0.002858 revset #2: not public() and branch("default") plain min max first last reverse 0) 0.012104 0.011138 0.011189 0.011138 0.011166 0.011578 1) 0.011387 94% 0.011738 105% 0.014220 127% 0.011223 0.011184 0.012077 revset #3: (not public() - obsolete()) plain min max first last reverse 0) 0.000583 0.000556 0.000552 0.000555 0.000552 0.000610 1) 0.000613 105% 0.000559 0.000557 0.000573 0.000558 0.000613 revset #4: head() - public() plain min max first last reverse 0) 0.010869 0.010800 0.011547 0.010843 0.010891 0.010891 1) 0.011031 0.011497 106% 0.011087 0.011100 0.011100 0.011085

contrib: clean up all-revsets.txt file I forgot to cleanup a handful of them when I originally created the file.

hgweb: link to revision by node hash in paper & coal Unlike other styles, paper and coal had only one link to current revision: in the sidebar. Since those links now use symbolic revisions after 3bb6f5f478a7, it's nice to have a link that allows going from /rev/tip to /rev/<tip hash>, for instance. Let's make the node hash in the page header that new link.

hgweb: link to revision by node hash in gitweb & monoblue This allows going from /rev/tip to /rev/<tip hash> with ease.

archive: report the node as "{p1node}+" when archiving a dirty wdir() This is more useful than reporting all 'f's, allowing the archive to be diffed against a specific revision to see what changed.

phase: remove a 'for x in "foo bar".split()' idiom in phasecache.replace I find this idiom fairly horrible.

phase: document the replace method This is a minor documentation update to answer a co-worker question.

revset: ensure we have loaded phases data in '_notpublic()' If we are the very first rev access (or if the phase cache just got invalidated) the phasesets will be None even if we support the native computation. So we explicitly trigger a computation if needed. This was not an issue before because requesting any phase information would have triggered such computation.

phase: rename getphaserevs to loadphaserevs This function is: - already loading the data in place, - used once in the code. So we drop the return value and change the name to make this obvious. We keep the function public because we'll have to use it in revset.

test-subrepo-recursion: more aggressively regex 'unzip -l' output for 10.10 The output on 10.10 looks like this: Length Date Time Name -------- ---- ---- ---- 172 01-01-80 00:00 .hg_archival.txt 10 01-01-80 00:00 .hgsub 45 01-01-80 00:00 .hgsubstate 3 01-01-80 00:00 x.txt 10 01-01-80 00:00 foo/.hgsub 45 01-01-80 00:00 foo/.hgsubstate 9 01-01-80 00:00 foo/y.txt 9 01-01-80 00:00 foo/bar/z.txt -------- ------- 303 8 files (2 digit year, shorter -- separators and closer columns). We don't care about any of that, so ignore it.

contrib: introduce an all-revsets.txt file This file should gather all revsets ever thought interesting by anyone. That way one can check the impact of a change when touching something revset-ish. See inline comments for details. This file have been refilled with all the entry I could automatically find from changeset descriptions. I assume we missed some not using 'revsetbenchmarks.py' output.

contrib: rename revsetbenchmarks.txt to 'base-revsets.txt' We rename the file and document its purpose. We'll be introducing another file gathering revsets useful for benchmark of the predicate themsleves in a coming changesets.

revsetbenchmarks: add main documention for the script This allow us to document the fact we can use comment in the file listing revsets.

hgweb: don't dereference symbolic revision in paper & coal style (issue2296) Let's make paper (and coal, since it borrows so much from paper) templates use symbolic revision in navigation links. The majority of links (log, filelog, annotate, etc) still use node hashes. Some pages don't have permanent links to current node hash (so it's not very easy to go from /rev/tip to /rev/<tip hash>), this will be addressed in future patches.

hgweb: don't dereference symbolic revision in gitweb style Let's make gitweb templates use symbolic revision in navigation links. The majority of links (log, filelog, annotate, etc) still use node hashes. Some pages don't have permanent links to current node hash (so it's not very easy to go from /rev/tip to /rev/<tip hash>), this will be addressed in future patches.

hgweb: don't dereference symbolic revision in monoblue style Let's make monoblue templates use symbolic revision in navigation links. The majority of links (log, filelog, annotate, etc) still use node hashes. Some pages don't have permanent links to current node hash (so it's not very easy to go from /rev/tip to /rev/<tip hash>), this will be addressed in future patches.

hgweb: don't dereference symbolic revision in spartan style Let's make spartan templates use symbolic revision in navigation links. The majority of links (log, filelog, annotate, etc) still use node hashes, and many pages also have permanent link to current node hash (i.e. you can go from /rev/tip to /rev/<tip hash> without manual url editing), so it's safe to update navigation.

hgweb: provide symrev (symbolic revision) property to the templates One of the features of hgweb is that current position in repo history is remembered between separate requests. That is, links from /rev/<node_hash> lead to /file/<node_hash> or /log/<node_hash>, so it's easy to dig deep into the history. However, such links could only use node hashes and local revision numbers, so while staying at one exact revision is easy, staying on top of the changes is not, because hashes presumably can't change (local revision numbers can, but probably not in a way you'd find useful for navigating). So while you could use 'tip' or 'default' in a url, links on that page would be permanent. This is not always desired (think /rev/tip or /graph/stable or /log/@) and is sometimes just confusing (i.e. /log/<not the tip hash>, when recent history is not displayed). And if user changed url deliberately to say default instead of <some node hash>, the page ignores that fact and uses node hash in its links, which means that navigation is, in a way, broken. This new property, symrev, is used for storing current revision the way it was specified, so then templates can use it in links and thus "not dereference" the symbolic revision. It is an additional way to produce links, so not every link needs to drop {node|short} in favor of {symrev}, many will still use node hash (log and filelog entries, annotate lines, etc). Some pages (e.g. summary, tags) always use the tip changeset for their context, in such cases symrev is set to 'tip'. This is needed in case the pages want to provide archive links. highlight extension needs to be updated, since _filerevision now takes an additional positional argument (signature "web, req, tmpl" is used by most of webcommands.py functions). More references to symbolic revisions and related gripes: issue2296, issue2826, issue3594, issue3634.

archive: support 'wdir()' This is a step toward replacing the extdiff internals with archive, in order to support 'extdiff -S'. Only Mercurial subrepos are supported for now. If a file is missing from the filesystem, it is silently skipped. Perhaps it should warn, but it cannot abort when working with extdiff because deleting a file is a legitimate diff.

subrepo: allow a representation of the working directory subrepo Some code cannot handle a subrepo based on the working directory (e.g. sub.dirty()), so the caller must opt in. This will be useful for archive, and perhaps some other commands. The git and svn methods where this is used may need to be fixed up on a case by case basis.

templater: comment that gettemplate() has different name resolution order I've tried to unify gettemplate() with buildtemplate(), but it didn't go well because gettemplate() have to bypass mapping dict. For example, web templates have '{tags%changelogtag}' and 'changelogtag' is defined in both mapping, the default, and context.cache, sourced from map file. In general, mapping shadows context variables, but gettemplate() have to pick it from context.cache.

templater: drop strtoken argument from compiletemplate() There's no "rawstring" template now.

templater: do not reevaluate rawstring as template (BC) The previous patch made 'string' is always interpreted as a template. So this patch removes the special handling of r'rawstring' instead. Now r'' disables template processing at all.

templater: take any string literals as template, but not for rawstring (BC) This patch series is intended to unify the interpretation of string literals. It is breaking change that boldly assumes a. string literal "..." never contains template-like fragment or it is intended to be a template b. we tend to use raw string literal r"..." for regexp pattern in which "{" should have different meaning Currently, we don't have a comprehensible rule how string literals are evaluated in template functions. For example, fill() takes "initialindent" and "hangindent" as templates, but not for "text", whereas "text" is a template in pad() function. date(date, fmt) diff(includepattern, excludepattern) fill(text, width, initialident: T, hangindent: T) get(dict, key) if(expr, then: T, else: T) ifcontains(search, thing, then: T, else: T) ifeq(expr1, expr2, then: T, else: T) indent(text, indentchars, firstline) join(list, sep) label(label: T, expr: T) pad(text: T, width, fillchar, right) revset(query, formatargs...]) rstdoc(text, style) shortest(node, minlength) startswith(pattern, text) strip(text, chars) sub(pattern, replacement, expression: T) word(number, text, separator) expr % template: T T: interpret "string" or r"rawstring" as template This patch series adjusts the rule as follows: a. string literal, '' or "", starts template processing (BC) b. raw string literal, r'' or r"", disables both \-escape and template processing (BC, done by subsequent patches) c. fragment not surrounded by {} is non-templated string "ccc{'aaa'}{r'bbb'}" ------------------ *: template --- c: string --- a: template --- b: rawstring Because this can eliminate the compilation of template arguments from the evaluation phase, "hg log -Tdefault" gets faster. % cd mozilla-central % LANG=C HGRCPATH=/dev/null hg log -Tdefault -r0:10000 --time > /dev/null before: real 4.870 secs (user 4.860+0.000 sys 0.010+0.000) after: real 3.480 secs (user 3.440+0.000 sys 0.030+0.000) Also, this will allow us to parse nested templates at once for better error indication.

templater: move runtemplate function out of buildmap/runmap pair The next patch will introduce buildtemplate function that should be defined near runtemplate. But I don't want to insert it between buildmap and runmap.

phase: also overwrite phase's sets when replacing a phasecache We need to copy this new attributes around too. This fix an issue where phases data used by 'not public()' were not invalidated properly.

phase: invalidate the phase's set cache alongside the revs Invalidate was leaving set data around leading to possible bugs in revset.

phase: also copy phase's sets when copying phase cache We forgot to add such copy when we added the attributes.

verify: check the subrepository references in .hgsubstate While hopefully atypical, there are reasons that a subrepository revision can be lost that aren't covered by corruption of the .hgsubstate revlog. Such things can happen when a subrepo is amended, stripped or simply isn't pulled from upstream because the parent repo revision wasn't updated yet. There's no way to know if it is an error, but this will find potential problems sooner than when some random revision is updated. Until recently, convert made no attempt at rewriting the .hgsubstate file. The impetuous for this is to verify the conversion of some repositories, and this is orders of magnitude faster than a bash script from 0..tip that does an 'hg update -C $rev'. But it is equally useful to determine if everything has been pulled down before taking a thumb drive on the go. It feels somewhat wrong to leave this out of verifymod (mostly because the file is already read in there, and the final summary is printed before the subrepos are checked). But verifymod looks very low level, so importing subrepo stuff there seems more wrong.

context: override workingctx.hex() to avoid a crash Since node is None for workingctx, it can't use the base class implementation of 'hex(self.node())'. It doesn't appear that there are any current callers of this, but there will be when archive supports 'wdir()'. My first thought was to use "{p1node}+", but that would cause headaches elsewhere [1]. We should probably fix up localrepository.__getitem__ to accept this hash for consistency, as a followup. This works, if the full hash is specified: @@ -480,7 +480,7 @@ return dirstate.dirstate(self.vfs, self.ui, self.root, validate) def __getitem__(self, changeid): - if changeid is None: + if changeid is None or changeid == 'ff' * 20: return context.workingctx(self) if isinstance(changeid, slice): return [context.changectx(self, i) That differs from null, where it will accept any number of 0s, as long as it isn't ambiguous. [1] https://www.selenic.com/pipermail/mercurial-devel/2015-June/071166.html

convert: update 'intermediate-source' in the destination's extras dictionary

check-code: ban use of '[[ ]]' in tests

test-fileset: remove bashism, use test instead of '[[ ]]' Debian dash complains about it. $TESTTMP.sh: 213: $TESTTMP.sh: [[: not found

tests: test symbolic revision (de)reference in all hgweb styles Right now the way revisions get specified in hgweb urls is ignored, i.e. after revision is resolved, only its node hash (or sometimes local revision number) is used for all links in the templates. So, basically, every page for "tip" revision (or any other symbolic rev id) will dereference it: lose the nice symbolic name by putting node hash/local rev number in its place. The only exception so far is archive links on some pages: /archive/tip.{bz2,gz,zip}. The fact that this dereferencing is neither convenient nor intuitive is reflected in issue2296, issue2826 and issue3594. issue3634 also mentions this. But to fix this it's first needed to demonstrate and test the way templates currently form links. The new test file is separate from other hgweb tests, since it seems big and distinct enough. And it's so big because links are formed in each template independently, so it's necessary to test them all to avoid any inconsistent behavior.

dirstate: use a presized dict for the dirstate This uses a simple heuristic to avoid expensive resizes. On a real-world repo with around 400,000 files, perfdirstate: before: ! wall 0.155562 comb 0.160000 user 0.150000 sys 0.010000 (best of 64) after: ! wall 0.132638 comb 0.130000 user 0.120000 sys 0.010000 (best of 75) On another real-world repo with around 250,000 files: before: ! wall 0.098459 comb 0.100000 user 0.090000 sys 0.010000 (best of 100) after: ! wall 0.089084 comb 0.090000 user 0.080000 sys 0.010000 (best of 100)

parsers: add an API to create a new presized dict

parsers: factor out code to create a presized dict In upcoming patches we'll expose this as an API.

parsers: drop Python 2.4 compat in make_file_foldmap

progress: respect ui.quiet (issue4726) Caught when I was running the hgsubversion testsuite and it started printing progress bars over top of my test output lines.

merge with stable

match: don't remove '.' from _includeroots This makes _includeroots more like _fileroots and gives visitdir() a nice symmetry in the two. I'm hoping to later combine the two (_fileroots and _includeroots), and having them treated similarly should make that step easier to follow.

match: join two nested if-blocks Instead of if a: if b: return False let's write it if a and b: return False

match: drop optimization (?) of 'parentdirs' calculation It seems unlikely that the optimization to avoid calling util.finddirs twice will be noticeable, so let's drop it. This makes the two conditions for includes and regular patterns more similar.

match: break boolean expressions into one operand per line This makes it much easier to spot both the operators ('and'/'or') and the operands.

match: drop unnecessary removal of '.' from excluded roots The repo root is nothing special when it comes to what directories to visit: patterns like '-X relglob:*.py' should not exclude the top directory, while '-X path:.' should (pointless as such a pattern may be). The explicit removal of '.' from the set of excluded roots was probably there to avoid removing the the root directory when any patterns had been given, but since 20ad936ac5d2 (treemanifest: visit directory 'foo' when given e.g. '-X foo/ba?', 2015-05-27), we only exclude directories that should be completely excluded, so we no longer need to special-case the root directory.