changegroup: compute seen files as changesets are added (issue4750) Before this patch, addchangegroup() would walk the changelog and compute the set of seen files between applying changesets and applying manifests. When cloning large repositories such as mozilla-central, this consumed a non-trivial amount of time. On my MBP, this walk takes ~10s. On a dainty EC2 instance, this was measured to take ~125s! On the latter machine, this delay was enough for the Mercurial server to disconnect the client, thinking it had timed out, thus causing a clone to abort. This patch enables the changelog to compute the set of changed files as new revisions are added. By doing so, we: * avoid a potentially heavy computation between changelog and manifest processing by spreading the computation across all changelog additions * avoid extra reads from the changelog by operating on the data as it is added The downside of this is that the add revision callback does result in extra I/O. Before, we would perform a flush (and subsequent read to construct the full revision) when new delta chains were created. For changelogs, this is typically every 2-4 revisions. Using the callback guarantees there will be a flush after every added revision *and* an open + read of the changelog to obtain the full revision in order to read the added files. So, this increases the frequency of these operations by the average chain length. In the future, the revlog should be smart enough to know how to read revisions that haven't been flushed yet, thus eliminating this extra I/O. On my MBP, the total CPU times for an `hg unbundle` with a local mozilla-central gzip bundle containing 251,934 changesets and 211,065 files did not have a statistically significant change with this patch, holding steady around 360s. So, the increased revlog flushing did not have an effect. With this patch, there is no longer a visible pause between applying changeset and manifest data. Before, it sure felt like Mercurial was lethargic making this transition. Now, the transition is nearly instantaneous, giving the impression that Mercurial is faster. Of course, eliminating this pause means that the potential for network disconnect due to channel inactivity during the changelog walk is eliminated as well. And that is the impetus behind this change.

revlog: add support for a callback whenever revisions are added A subsequent patch will add a feature that performs iterative computation as changesets are added from a changegroup. To facilitate this type of processing in a generic manner, we add a mechanism for calling a function whenever a revision is added via revlog.addgroup(). There are potential performance concerns with this callback, as using it will flush the revlog after every revision is added.

crecord: throws error instead of crashing for large diffs Before this patch, crecord was crashing for large diffs (30k lines on my laptop). This patch catches the exception raised in that case and use the error reporting mechanism introduced in the previous patch for notifying the user of the issue. It is not possible to add a test for that for now as we don't yet have full blown ui tests for the curses interface.

crecord: add error reporting for failure in curses interface initialization Before this patch, we couldn't report to the user any error that occurred: - after we enabled the curses interface but - before the interface is set up and drawn This patch, provides a way to set errors that happens during the initialization of the interface and log them once the curses interface has been displayed.

revset: parse nullary ":" operator as "0:tip" This is necessary for compatibility with the old-style parser that will be removed by future patches.

parser: take suffix action if no infix action is defined If no infix action is defined, a suffix action isn't ambiguous, so it should be taken no matter if the next token can be an operand. This is exactly the same flow as prefix/primary handling. This change has no effect now because all suffix tokens have infix actions.

parser: reorder infix/suffix handling to be similar to prefix/primary flow It can be exactly the same flow as the prefix/primary handling. A suffix action is accepted only if the next token never starts new term.

parser: resolve ambiguity where both prefix and primary actions are defined If both actions are defined, a primary-expression action is accepted only if the next token never starts new term. For example, parsed as primary expression: ":" # next token 'end' has no action "(:)" # next token ')' has no action ":+y" # next token '+' is infix operator parsed as prefix operator: ":y" # next token 'y' is primary expression ":-y" # next token '-' is prefix operator This is mostly the same resolution as the infix/suffix rules.

parser: separate actions for primary expression and prefix operator This will allow us to define both a primary expression, ":", and a prefix operator, ":y". The ambiguity will be resolved by the next patch. Prefix actions in elements table are adjusted as follows: original prefix primary prefix ----------------- -------- ----------------- ("group", 1, ")") -> n/a ("group", 1, ")") ("negate", 19) -> n/a ("negate", 19) ("symbol",) -> "symbol" n/a

changelog: update read pending documentation The pending index contains a full copy of the index + in-transaction data. We replace "extend" with "overwrite" to make this clearer.

extdiff: add support for subrepos Git and svn subrepo support is incomplete, because they don't support archiving the working copy.

extdiff: use archiver to take snapshots of committed revisions This is the last step before supporting extdiff -S. It maintains the existing behavior of diffing the largefile standins instead of the largefiles themselves. Note however that the standins are not updated immediately upon modification, so uncommitted largefile changes are ignored, as they previously were, even with the diff command.

largefiles: allow the archiving of largefiles to be disabled There are currently no users of this, but it is a necessary step before converting extdiff to use archive. It may be useful to add an argument to extdiff in the future and allow largefiles to be diffed, but archiving largefiles can have significant overhead and may not be very diffable, so archiving them by default seems wrong. It is a mystery to me why the lfstatus attribute needs to be set on the unfiltered repo. However if it is set on the filtered repo instead (and the filtered repo is passed to the original command), the lfstatus attribute is False in the overrides for archival.archive() and hgsubrepo.archive() when invoking the archive command. This smells like the buggy status behavior (see 67d63ec85eb7, which was reverted in df463ca0adef). Neither the status nor summary commands have this weird behavior in their respective overrides.

parsers: fix buffer overflow by invalid parent revision read from revlog If revlog file is corrupted, it can have parent pointing to invalid revision. So we should validate it before updating nothead[], phases[], seen[], etc. Otherwise it would cause segfault at best. We could use "rev" instead of "maxrev" as upper bound, but I think the explicit "maxrev" can clarify that we just want to avoid possible buffer overflow vulnerability.

histedit: mark temporary commits as obsolete when allowed to Before this patch, we were stripping temporary commits at the end of a histedit whether it was successful or not. If we can create obs markers, we should create them instead of stripping because it is faster and safer.

histedit: minor refactoring of createmarkers check We use a variable to store whether or not we can create obsolescence markers. It makes the patch series more readable as we are going to reuse this values in other places in the function.

crecord: fix issue when backgrounding editor would leave artefact Before this patch: - if a user was entering a commit message after having ran the curses interface - and then uses ctrl-z, followed by fg to put the editor in the background/foreground - then the curses interface would leave artefact on the screen of the editor, making entering the commit message a difficult task This happened because ncurses registers a signal handler for SIGTSTP and does not restore the original signal handler after running. More info at: http://stackoverflow.com/questions/31440392/ curses-wrapper-messing-up-terminal-after-background-foreground-sequence/ 31441709#31441709 This patch restores the original value of the signal handler after running the curses interface and therefore fixes this issue. It don't know how to add a test for this issue, I tested the scenario above manually and it works correctly with the patch.

censor: make various path forms available like other Mercurial commands Before this patch, censored file should be exactly "a path relative to repository root" regardless of current working directory, because "hg censor" passes "path" to "repo.file()" directly without any preparations. To make various path forms available like other Mercurial commands, this patch gets a target file path in the way of "hg parents FILE". Getting "wctx" is relocated to reuse "wctx" for efficiency.

convert: use 'default' for specifying branch name in branchmap (issue4753) A fix for issue2653 with 1d155582a8ea introduced a discrepancy how default branch should be denoted when converting with branchmap from different SCM. E.g. for Git and Mercurial you need to use 'default' whilst for Perforce and SVN you had to use 'None'. This changeset unifies 'default' for such purposes whilst falling back to 'None' when no 'default' mapping specified.

parser: extract function that tests if next token may start new term Future patches will separate primary expression and prefix operator actions. This function will be used to resolve ambiguity of them. This is a step to remove the old-style revexpr parser. We need both ":" and ":y" operators for backward compatibility.

parser: factor out function that parses right-hand side of prefix/infix ops These two had common pattern. The significant difference was just a result expression: prefix: (op-name, rhs) infix: (op-name, lhs, rhs)

parser: remove unused parameter 'pos' from _match() This backs out 9d1cf337a78d. The issue spotted by that changeset was addressed earlier by d4cafcb63f77.

parser: fill invalid infix and suffix actions by None This can simplify the expansion of (prefix, infix, suffix) actions.

parser: add comment about structure of elements to each table

shelve: omit incorrect 'commit' suggestion at 'hg shelve -i' Before this patch, 'hg shelve -i' under non-interactive mode suggests 'use commit instead', and it obviously incorrect, because what user wants to do isn't 'commit' but 'shelve'. To omit incorrect 'commit' suggestion at 'hg shelve -i', this patch specifies 'None' for 'cmdsuggest' argument of 'cmdutil.dorecord()'.

record: omit meaningless 'qrefresh' suggestion at 'hg qrefresh -i' Before this patch, 'hg qrefresh -i' under non-interactive mode suggests 'use qrefresh instead', and it obviously meaningless. To omit meaningless 'qrefresh' suggestion at 'hg qrefresh -i', this patch specifies 'None' for 'cmdsuggest' argument of 'cmdutil.dorecord()'.

record: omit meaningless 'qnew' suggestion at 'hg qnew -i' Before this patch, 'hg qnew -i' under non-interactive mode suggests 'use qnew instead', and it obviously meaningless. To omit meaningless 'qnew' suggestion at 'hg qnew -i', this patch adds internal function '_qrecord()' and specifies 'cmdsuggest' for each of 'qrecord' and 'qnew' separately.

record: omit meaningless 'commit' suggestion at 'hg commit -i' Before this patch, 'hg commit -i' under non-interactive mode suggests 'use commit instead', and it obviously meaningless. This patch makes 'record.record'()' examine 'ui.interactive()' and show suggestion by itself before calling 'commands.commit()'. This allows 'commands.commit()' to specify 'None' for 'cmdsuggest' argument of 'cmdutil.dorecord()' to omit meaningless 'commit' suggestion at 'hg commit -i'.

cmdutil: allow callers of cmdutil.dorecord to omit suggestion Interactive committing under non-interactive mode shows command suggestion, but sometimes it is meaningless. command suggestion usability ------------ ---------- ----------- record commit commit -i commit meaningless qrecord qnew qnew -i qnew meaningless qrefersh -i qrefresh meaningless shelve -i commit incorrect ------------ ---------- ----------- This patch allows callers of 'cmdutil.dorecord()' to omit meaningless suggestion by passing None or so for 'cmdsuggest' argument of it. This is a preparation for subsequent patches, which fix each suggestion issues above.

spartan: don't drop current revision in log/graph links Just to be consistent with log and shortlog links, graph links should have a revision context too. And the same goes for the graph page, where it's log and shortlog links that should have context.

hgk: tweak doc format for path option This lets the config checker see it.

acl: mark deprecated config option This option has been undocumented since day 0.

gpg: mention undocumented options

check-config: add config option checker This script scans files for lines that look like either ui.config usage or config variable documentation. It then ensures: - ui.config calls for each option agree on types and defaults - every option appears to be mentioned in documentation It doesn't complain about devel/experimental options and allows marking options that are not intended to be public. Since we haven't been able to come up with a good scheme for documenting config options at point of use, this will help close the loop of making sure all options that should be documented are.

bookmarks: clear active bookmark on non-linear update

convert: unescape Perforce-escaped special characters in filenames

convert: allow customizing git remote prefix Previously all git remotes were created as "remote/foo". This patch adds a configuration option for deciding what the prefix should be. This is useful if you want the bookmarks to be "origin/foo" like they are in git, or if you're integrating with the remotenames extension and don't want the local remote/foo bookmarks to overlap with the remote foo bookmarks.

help: rewrite template examples to not use shell escaping Though they work fine even on cmd.exe, these examples could lead to misguided understanding about the template syntax.

templater: unify "string" and "rawstring" "rawstring" was introduced by 5ab28a2e9962, but it's no longer necessary because c1975809a6b5 and fd5bc660c9f0 changed the way of processing string literals. This patch moves string decoding to the parsing phase as it was before: ('rawstring', s) -> ('string', s) ('string', s) -> ('string', s.decode('string-escape'))

templater: remove processing of "string" literals from tokenizer They are processed as "template" strings now.

templater: introduce one-pass parsing of nested template strings Instead of re-parsing quoted strings as templates, the tokenizer can delegate the parsing of nested template strings to the parser. It has two benefits: 1. syntax errors can be reported with absolute positions 2. nested template can use quotes just like shell: "{"{rev}"}" It doesn't sound nice that the tokenizer recurses into the parser. We could instead make the tokenize itself recursive, but it would be much more complicated because we would have to adjust binding strengths carefully and put dummy infix operators to concatenate template fragments. Now "string" token without r"" never appears. It will be removed by the next patch.

templater: check existence of closing brace of template string

templater: extract function that parses template string It will be called recursively to parse nested template strings.

templater: respect stop position while parsing template string It has no effect now because stop is len(tmpl).

hgweb: provide links to branches, tags and bookmarks by name (paper and coal) It's sometimes handy to, say, have a url always point to branch head, not just at the current branch head by node hash. Previously, this was only possible by manually editing url and replacing node hash with branch/tag/bookmark name. It wasn't very convenient, or easy - in case the name contained special characters that needed to be urlencoded. Let's have /branches, /tags and /bookmarks pages in paper and coal style provide links both to symbolic revisions and to node hashes. This feature was wished for in issue3594.

templates: introduce revescape filter for escaping symbolic revisions There needs to be a way to escape symbolic revisions containing forward slashes, but urlescape filter doesn't escape slashes at all (in fact, it is used in places where forward slashes must be preserved). The filter considers @ to be safe just for bookmarks like @ and @default to look good in urls.

hgweb: allow symbolic revisions with forward slashes in urls It's possible to have a branch/tag/bookmark with all kinds of special characters, such as {}/\!?. While not very conveniently, symbolic revisions with such characters work from command line if user correctly quotes the characters. These characters also work in hgweb, when they are properly encoded, with one exception: '/' (forward slash, urlencoded as '%2F'), which was getting decoded before hgweb could parse it as a part of PATH_INFO. Because of that, hgweb was seeing it as any other forward slash, that is, as just another url parts separator. For example, if user wanted to see the content of dir/file at bookmark 'feature/eggs', url could be: '/file/feature%2Feggs/dir/file'. But hgweb tried to find a revision 'feature' and get contents of 'eggs/dir/file'. To fix this, let's assume forward slashes are doubly-urlencoded (%252F), so CGI/WSGI server decodes it into %2F. Then we can decode %2F in the revision part of the url into an actual '/' character. Making hgweb produce such urls will be done in the next 2 patches.

convert: ignore case changes in vieworder for Perforce Perforce sometimes mixes the case resulting in files being ignored.

convert: if getting a file from Perforce fails try to get it one more time When converting a particularly large Perforce changelist (especially with some big files), it is very likely to run into an intermittent network issue (e.g. WSAECONNRESET or WSAETIMEDOUT) getting one of the files, which will result in the entire changelist converting being aborted. Which can be quite unfortunate since you might have waited hours getting all other files. To mitigate this let's attempt to get the file one more time, escalating original exception if that attempt fails.

shelve: keep old backups if timestamp can't decide exact order of them Before this patch, backups to be discarded are decided by steps below at 'hg unshelve' or so: 1. list '(st_mtime, filename)' tuples of each backups up 2. sort list of these tuples, and 3. discard backups other than 'maxbackups' ones at the end of list This doesn't work well in the case below: - "sort by name" order differs from actual backup-ing order, and - some of backups have same timestamp For example, 'test-shelve.t' satisfies the former condition: - 'default-01' < 'default-1' in "sort by name" order - 'default-1' < 'default-01' in actual backup-ing order Then, 'default-01' is discarded instead of 'default-1' unexpectedly, if they have same timestamp. This failure appears occasionally, because the most important condition "same timestamp" is timing critical. To avoid such unexpected discarding, this patch keeps old backups if timestamp can't decide exact order of them. Timestamp of the border backup (= the oldest one of recent 'maxbackups' ones) as 'bordermtime' is used to examine whether timestamp can decide exact order of backups.

subrepo: use vfs.dirname instead of os.path.dirname This patch uses "wvfs of the parent repository" ('pwvfs') instead of 'wvfs' of own repository, because 'self._path' is the path to this subrepository as seen from the parent repository.

vfs: add dirname

subrepo: use vfs.basename instead of os.path.basename

vfs: add basename

subrepo: use repo.pathto instead of util.pathto to simplify invocation This centralization into 'repo.pathto()' should reduce the cost of vfs migration around 'getcwd()' and so on in the future.

subrepo: prefetch ctx.repo() for efficiency and centralization 'subrepo.state()' refers same 'ctx.repo()' in many places and times.