censor: make various path forms available like other Mercurial commands Before this patch, censored file should be exactly "a path relative to repository root" regardless of current working directory, because "hg censor" passes "path" to "repo.file()" directly without any preparations. To make various path forms available like other Mercurial commands, this patch gets a target file path in the way of "hg parents FILE". Getting "wctx" is relocated to reuse "wctx" for efficiency.

convert: use 'default' for specifying branch name in branchmap (issue4753) A fix for issue2653 with 1d155582a8ea introduced a discrepancy how default branch should be denoted when converting with branchmap from different SCM. E.g. for Git and Mercurial you need to use 'default' whilst for Perforce and SVN you had to use 'None'. This changeset unifies 'default' for such purposes whilst falling back to 'None' when no 'default' mapping specified.

parser: extract function that tests if next token may start new term Future patches will separate primary expression and prefix operator actions. This function will be used to resolve ambiguity of them. This is a step to remove the old-style revexpr parser. We need both ":" and ":y" operators for backward compatibility.

parser: factor out function that parses right-hand side of prefix/infix ops These two had common pattern. The significant difference was just a result expression: prefix: (op-name, rhs) infix: (op-name, lhs, rhs)

parser: remove unused parameter 'pos' from _match() This backs out 9d1cf337a78d. The issue spotted by that changeset was addressed earlier by d4cafcb63f77.

parser: fill invalid infix and suffix actions by None This can simplify the expansion of (prefix, infix, suffix) actions.

parser: add comment about structure of elements to each table

shelve: omit incorrect 'commit' suggestion at 'hg shelve -i' Before this patch, 'hg shelve -i' under non-interactive mode suggests 'use commit instead', and it obviously incorrect, because what user wants to do isn't 'commit' but 'shelve'. To omit incorrect 'commit' suggestion at 'hg shelve -i', this patch specifies 'None' for 'cmdsuggest' argument of 'cmdutil.dorecord()'.

record: omit meaningless 'qrefresh' suggestion at 'hg qrefresh -i' Before this patch, 'hg qrefresh -i' under non-interactive mode suggests 'use qrefresh instead', and it obviously meaningless. To omit meaningless 'qrefresh' suggestion at 'hg qrefresh -i', this patch specifies 'None' for 'cmdsuggest' argument of 'cmdutil.dorecord()'.

record: omit meaningless 'qnew' suggestion at 'hg qnew -i' Before this patch, 'hg qnew -i' under non-interactive mode suggests 'use qnew instead', and it obviously meaningless. To omit meaningless 'qnew' suggestion at 'hg qnew -i', this patch adds internal function '_qrecord()' and specifies 'cmdsuggest' for each of 'qrecord' and 'qnew' separately.

record: omit meaningless 'commit' suggestion at 'hg commit -i' Before this patch, 'hg commit -i' under non-interactive mode suggests 'use commit instead', and it obviously meaningless. This patch makes 'record.record'()' examine 'ui.interactive()' and show suggestion by itself before calling 'commands.commit()'. This allows 'commands.commit()' to specify 'None' for 'cmdsuggest' argument of 'cmdutil.dorecord()' to omit meaningless 'commit' suggestion at 'hg commit -i'.

cmdutil: allow callers of cmdutil.dorecord to omit suggestion Interactive committing under non-interactive mode shows command suggestion, but sometimes it is meaningless. command suggestion usability ------------ ---------- ----------- record commit commit -i commit meaningless qrecord qnew qnew -i qnew meaningless qrefersh -i qrefresh meaningless shelve -i commit incorrect ------------ ---------- ----------- This patch allows callers of 'cmdutil.dorecord()' to omit meaningless suggestion by passing None or so for 'cmdsuggest' argument of it. This is a preparation for subsequent patches, which fix each suggestion issues above.

spartan: don't drop current revision in log/graph links Just to be consistent with log and shortlog links, graph links should have a revision context too. And the same goes for the graph page, where it's log and shortlog links that should have context.

hgk: tweak doc format for path option This lets the config checker see it.

acl: mark deprecated config option This option has been undocumented since day 0.

gpg: mention undocumented options

check-config: add config option checker This script scans files for lines that look like either ui.config usage or config variable documentation. It then ensures: - ui.config calls for each option agree on types and defaults - every option appears to be mentioned in documentation It doesn't complain about devel/experimental options and allows marking options that are not intended to be public. Since we haven't been able to come up with a good scheme for documenting config options at point of use, this will help close the loop of making sure all options that should be documented are.

bookmarks: clear active bookmark on non-linear update

convert: unescape Perforce-escaped special characters in filenames

convert: allow customizing git remote prefix Previously all git remotes were created as "remote/foo". This patch adds a configuration option for deciding what the prefix should be. This is useful if you want the bookmarks to be "origin/foo" like they are in git, or if you're integrating with the remotenames extension and don't want the local remote/foo bookmarks to overlap with the remote foo bookmarks.

help: rewrite template examples to not use shell escaping Though they work fine even on cmd.exe, these examples could lead to misguided understanding about the template syntax.

templater: unify "string" and "rawstring" "rawstring" was introduced by 5ab28a2e9962, but it's no longer necessary because c1975809a6b5 and fd5bc660c9f0 changed the way of processing string literals. This patch moves string decoding to the parsing phase as it was before: ('rawstring', s) -> ('string', s) ('string', s) -> ('string', s.decode('string-escape'))

templater: remove processing of "string" literals from tokenizer They are processed as "template" strings now.

templater: introduce one-pass parsing of nested template strings Instead of re-parsing quoted strings as templates, the tokenizer can delegate the parsing of nested template strings to the parser. It has two benefits: 1. syntax errors can be reported with absolute positions 2. nested template can use quotes just like shell: "{"{rev}"}" It doesn't sound nice that the tokenizer recurses into the parser. We could instead make the tokenize itself recursive, but it would be much more complicated because we would have to adjust binding strengths carefully and put dummy infix operators to concatenate template fragments. Now "string" token without r"" never appears. It will be removed by the next patch.

templater: check existence of closing brace of template string

templater: extract function that parses template string It will be called recursively to parse nested template strings.

templater: respect stop position while parsing template string It has no effect now because stop is len(tmpl).

hgweb: provide links to branches, tags and bookmarks by name (paper and coal) It's sometimes handy to, say, have a url always point to branch head, not just at the current branch head by node hash. Previously, this was only possible by manually editing url and replacing node hash with branch/tag/bookmark name. It wasn't very convenient, or easy - in case the name contained special characters that needed to be urlencoded. Let's have /branches, /tags and /bookmarks pages in paper and coal style provide links both to symbolic revisions and to node hashes. This feature was wished for in issue3594.

templates: introduce revescape filter for escaping symbolic revisions There needs to be a way to escape symbolic revisions containing forward slashes, but urlescape filter doesn't escape slashes at all (in fact, it is used in places where forward slashes must be preserved). The filter considers @ to be safe just for bookmarks like @ and @default to look good in urls.

hgweb: allow symbolic revisions with forward slashes in urls It's possible to have a branch/tag/bookmark with all kinds of special characters, such as {}/\!?. While not very conveniently, symbolic revisions with such characters work from command line if user correctly quotes the characters. These characters also work in hgweb, when they are properly encoded, with one exception: '/' (forward slash, urlencoded as '%2F'), which was getting decoded before hgweb could parse it as a part of PATH_INFO. Because of that, hgweb was seeing it as any other forward slash, that is, as just another url parts separator. For example, if user wanted to see the content of dir/file at bookmark 'feature/eggs', url could be: '/file/feature%2Feggs/dir/file'. But hgweb tried to find a revision 'feature' and get contents of 'eggs/dir/file'. To fix this, let's assume forward slashes are doubly-urlencoded (%252F), so CGI/WSGI server decodes it into %2F. Then we can decode %2F in the revision part of the url into an actual '/' character. Making hgweb produce such urls will be done in the next 2 patches.

convert: ignore case changes in vieworder for Perforce Perforce sometimes mixes the case resulting in files being ignored.

convert: if getting a file from Perforce fails try to get it one more time When converting a particularly large Perforce changelist (especially with some big files), it is very likely to run into an intermittent network issue (e.g. WSAECONNRESET or WSAETIMEDOUT) getting one of the files, which will result in the entire changelist converting being aborted. Which can be quite unfortunate since you might have waited hours getting all other files. To mitigate this let's attempt to get the file one more time, escalating original exception if that attempt fails.

shelve: keep old backups if timestamp can't decide exact order of them Before this patch, backups to be discarded are decided by steps below at 'hg unshelve' or so: 1. list '(st_mtime, filename)' tuples of each backups up 2. sort list of these tuples, and 3. discard backups other than 'maxbackups' ones at the end of list This doesn't work well in the case below: - "sort by name" order differs from actual backup-ing order, and - some of backups have same timestamp For example, 'test-shelve.t' satisfies the former condition: - 'default-01' < 'default-1' in "sort by name" order - 'default-1' < 'default-01' in actual backup-ing order Then, 'default-01' is discarded instead of 'default-1' unexpectedly, if they have same timestamp. This failure appears occasionally, because the most important condition "same timestamp" is timing critical. To avoid such unexpected discarding, this patch keeps old backups if timestamp can't decide exact order of them. Timestamp of the border backup (= the oldest one of recent 'maxbackups' ones) as 'bordermtime' is used to examine whether timestamp can decide exact order of backups.

subrepo: use vfs.dirname instead of os.path.dirname This patch uses "wvfs of the parent repository" ('pwvfs') instead of 'wvfs' of own repository, because 'self._path' is the path to this subrepository as seen from the parent repository.

vfs: add dirname

subrepo: use vfs.basename instead of os.path.basename

vfs: add basename

subrepo: use repo.pathto instead of util.pathto to simplify invocation This centralization into 'repo.pathto()' should reduce the cost of vfs migration around 'getcwd()' and so on in the future.

subrepo: prefetch ctx.repo() for efficiency and centralization 'subrepo.state()' refers same 'ctx.repo()' in many places and times.

revset: rename getkwargs() to getargsdict() This function was added recently at 48919d246a47, but its name was misleading because it processes both positional and keyword arguments.

revset: work around x:y range where x or y is wdir() All revisions must be contiguous in spanset, so we need the special case for the wdir revision.

revset: use integer representation of wdir() in revset This is the simplest way to handle wdir() revision in revset. None didn't work well because revset heavily depends on integer operations such as min(), max(), sorted(), x:y, etc. One downside is that we cannot do "wctx.rev() in set" because wctx.rev() is still None. We could wrap the result set by wdirproxyset that translates None to wdirrev, but it seems overengineered at this point. result = getset(repo, subset, tree) if 'wdir' in funcsused(tree): result = wdirproxyset(result) Test cases need the '(all() + wdir()) &' hack because we have yet to fix the bootstrapping issue of null and wdir.

localrepo: provide workingctx by integer revision This allows us to use the integer representation in revset. None doesn't work well while computing revset because revset heavily depends on and optimized for integer revisions. Still repo[wdirrev].rev() is None, which means the canonical form of the working-directory revision is None. This patch doesn't add the case for the wdirid because we can't handle short and ambiguous identifiers here. Perhaps, the wdirid will have to be handled in the changelog layer.

changeset_printer: change flush() to accept ctx instead of rev Because flush() is the function to write data buffered by show(ctx), flush(ctx) is more consistent than flush(rev). This makes sure that buffered header and hunk are always keyed by ctx.rev(). This patch will allow us to give an integer to the wdir while keeping wctx.rev() -> None.

changeset_printer: display wdirrev/wdirnode values for workingctx Because we want to eliminate "if"s in the default template, it makes sense to display wdirrev/wdirnode values for now. wdir() is still experimental, so the output of "log -r'wdir()'" may change in future.

hg: support for auto sharing stores when cloning Many 3rd party consumers of Mercurial have created wrappers to essentially perform clone+share as a single operation. This is especially popular in automated processes like continuous integration systems. The Jenkins CI software and Mozilla's Firefox release automation infrastructure have both implemented custom code that effectively perform clone+share. The common use case here is that clients want to obtain N>1 checkouts while minimizing disk space and network requirements. Furthermore, they often don't care that a clone is an exact mirror of a remote: they are simply looking to obtain checkouts of specific revisions. When multiple third parties implement a similar feature, it's a good sign that the feature is worth adding to the core product. This patch adds support for an easy-to-use clone+share feature. The internal "clone" function now accepts options to control auto sharing during clone. When the auto share mode is active, a store will be created/updated under the base directory specified and a new repository pointing to the shared store will be created at the path specified by the user. The share extension has grown the ability to pass these options into the clone command/function. No command line options for this feature are added because we don't feel the feature will be popular enough to warrant their existence. There are two modes for auto share mode. In the default mode, the shared repo is derived from the first changeset (rev 0) in the remote repository. This enables related repositories existing at different URLs to automatically use the same storage. In environments that operate several repositories (separate repo for branch/head/bookmark or separate repo per user), this has the potential to drastically reduce storage and network requirements. In the other mode, the name is derived from the remote's path/URL.

merge with stable

cmdutil: apply dirstate.normallookup on (maybe partially) committed files To detect change of a file without redundant comparison of file content, dirstate recognizes a file as certainly clean, if: (1) it is already known as "normal", (2) dirstate entry for it has valid (= not "-1") timestamp, and (3) mode, size and timestamp of it on the filesystem are as same as ones expected in dirstate This works as expected in many cases, but doesn't in the corner case that changing a file keeps mode, size and timestamp of it on the filesystem. The timetable below shows steps in one of typical such situations: ---- ----------------------------------- ---------------- timestamp of "f" ---------------- dirstate file- time action mem file system ---- ----------------------------------- ---- ----- ----- N *** *** - change "f" N - execute 'hg commit -i' - backup "f" with timestamp N - revert "f" by 'merge.update()' N with 'partially' - apply selected hunks N by 'patch.patch()' - 'repo.commit()' - 'dirstate.normal("f")' N N+1 - 'dirstate.write()' N N - restore "f" N+1 - restore timestamp of "f" N - 'hg status' shows "f" as "clean" N N N ---- ----------------------------------- ---- ----- ----- The most important point is that 'dirstate.write()' is executed at N+1 or later. This causes writing dirstate timestamp N of "f" out successfully. If it is executed at N, 'parsers.pack_dirstate()' replaces timestamp N with "-1" before actual writing dirstate out. This issue can occur when 'hg commit -i' satisfies conditions below: - the file is committed partially, and - mode and size of the file aren't changed before and after committing The root cause of this issue is that (maybe partially changed) files are restored with original timestamp but dirstate isn't updated for them. To detect changes of files correctly, this patch applies 'dirstate.normallookup()' on restored files. Status check is needed before 'dirstate.normallookup()', because status other than "n(ormal)" should be kept at failure of committing. This patch doesn't examine whether each files are committed fully or partially, because interactive hunk selection makes it difficult. After this change, timetable is changed as below: ---- ----------------------------------- ---------------- timestamp of "f" ---------------- dirstate file- time action mem file system ---- ----------------------------------- ---- ----- ----- N *** *** - change "f" N - execute 'hg commit -i' - backup "f" with timestamp N - revert "f" by 'merge.update()' N with 'partially' - apply selected hunks N by 'patch.internalpatch()' - 'repo.commit()' - 'dirstate.normal("f")' N N+1 - 'dirstate.write()' N N - restore "f" N+1 - restore timestamp of "f" N ----------------------------------- ---- ----- ----- - normallookup("f") -1 - release wlock - 'dirstate.write()' -1 -1 N ----------------------------------- ---- ----- ----- - 'hg status' shows "f" as "clean" -1 -1 N ---- ----------------------------------- ---- ----- ----- To reproduce this issue in tests certainly, this patch emulates some timing critical actions as below: - change "f" at N 'touch -t 200001010000' before command invocation changes mtime of "f" to "2000-01-01 00:00" (= N). - apply selected hunks at N 'patch.internalpatch()' with 'fakepatchtime.py' explicitly changes mtime of patched files to "2000-01-01 00:00" (= N). - 'dirstate.write()' at N+1 (or "not at N") 'pack_dirstate()' uses actual timestamp at runtime as "now", and it should be different from the "2000-01-01 00:00" of "f". BTW, in 'test-commit-interactive.t', files are sometimes treated as modified , even though they are just committed fully via 'hg commit -i' and 'hg diff' shows nothing for them. Enabling win32text causes EOL style mismatching below: - files are changed in LF style EOL => files restored after committing uses LF style EOL (1) - 'merge.update()' reverts files in CRLF style EOL - 'patch.internalpatch()' changes files in CRLF style EOL => 'dirstate.normal()' via 'repo.commit()' uses the size of files in CRLF style EOL (2) Therefore, fully committed files are treated as "modified", because 'lstat()' returns size of (1) restored files in LF style EOL, but dirstate expects size of (2) committed files in CRLF style EOL. After this patch, 'dirstate.normallookup()' on committed files forces subsequent 'hg status' to examine changes exactly, and fully committed files are treated as clean as expected. This is reason why this patch also does: - add some 'hg status' checking status of fully committed files - clear win32text configuration before size/timestamp sensitive examination

cmdutil: put recordfunc invocation into wlock scope for consistency Before this patch, 'recordfunc()' for interactive hunk selection does below outside wlock scope at 'hg commit -i' and so on: - backup files, which may be partially changed - apply selected hunks on files - restore files from backup-ed ones These should be executed inside wlock scope for consistency. To put them into wlock scope without largely changing indents in 'recordfunc()', this patch adds another wrapper function. This patch is also a preparation for subsequent patch fixing the issue to correctly recognize partially committed files as "modified".

context: write dirstate out explicitly at the end of markcommitted To detect change of a file without redundant comparison of file content, dirstate recognizes a file as certainly clean, if: (1) it is already known as "normal", (2) dirstate entry for it has valid (= not "-1") timestamp, and (3) mode, size and timestamp of it on the filesystem are as same as ones expected in dirstate This works as expected in many cases, but doesn't in the corner case that changing a file keeps mode, size and timestamp of it on the filesystem. The timetable below shows steps in one of typical such situations: ---- ----------------------------------- ---------------- timestamp of "f" ---------------- dirstate file- time action mem file system ---- ----------------------------------- ---- ----- ----- * *** *** - 'hg transplant REV1 REV2 ...' - transplanting REV1 .... N - change "f", but keep size N (via 'patch.patch()') - 'dirstate.normal("f")' N *** (via 'repo.commit()') - transplanting REV2 - change "f", but keep size N (via 'patch.patch()') - aborted while patching N+1 - release wlock - 'dirstate.write()' N N N - 'hg status' shows "r1" as "clean" N N N ---- ----------------------------------- ---- ----- ----- The most important point is that 'dirstate.write()' is executed at N+1 or later. This causes writing dirstate timestamp N of "f" out successfully. If it is executed at N, 'parsers.pack_dirstate()' replaces timestamp N with "-1" before actual writing dirstate out. This issue can occur when 'hg transplant' satisfies conditions below: - multiple revisions to be transplanted change the same file - those revisions don't change mode and size of the file, and - the 2nd or later revision of them fails after changing the file The root cause of this issue is that files are changed without flushing in-memory dirstate changes via 'repo.commit()' (even though omitting 'dirstate.normallookup()' on files changed by 'patch.patch()' for efficiency also causes this issue). To detect changes of files correctly, this patch writes in-memory dirstate changes out explicitly after marking files as clean in 'committablectx.markcommitted()', which is invoked via 'repo.commit()'. After this change, timetable is changed as below: ---- ----------------------------------- ---------------- timestamp of "f" ---------------- dirstate file- time action mem file system ---- ----------------------------------- ---- ----- ----- * *** *** - 'hg transplant REV1 REV2 ...' - transplanting REV1 .... N - change "f", but keep size N (via 'patch.patch()') - 'dirstate.normal("f")' N *** (via 'repo.commit()') ----------------------------------- ---- ----- ----- - 'dirsttate.write()' -1 -1 ----------------------------------- ---- ----- ----- - transplanting REV2 - change "f", but keep size N (via 'patch.patch()') - aborted while patching N+1 - release wlock - 'dirstate.write()' -1 -1 N - 'hg status' shows "r1" as "clean" -1 -1 N ---- ----------------------------------- ---- ----- ----- To reproduce this issue in tests certainly, this patch emulates some timing critical actions as below: - change "f" at N 'patch.patch()' with 'fakepatchtime.py' explicitly changes mtime of patched files to "2000-01-01 00:00" (= N). - 'dirstate.write()' via 'repo.commit()' at N 'fakedirstatewritetime.py' forces 'pack_dirstate()' to use "2000-01-01 00:00" as "now", only if 'pack_dirstate()' is invoked via 'committablectx.markcommitted()'. - 'dirstate.write()' via releasing wlock at N+1 (or "not at N") 'pack_dirstate()' via releasing wlock uses actual timestamp at runtime as "now", and it should be different from the "2000-01-01 00:00" of "f". BTW, this patch doesn't test cases below, even though 'patch.patch()' is used similarly in these cases: 1. failure of 'hg import' or 'hg qpush' 2. success of 'hg import', 'hg qpush' or 'hg transplant' Case (1) above doesn't cause this kind of issue, because: - if patching is aborted by conflicts, changed files are committed changed files are marked as CLEAN, even though they are partially patched. - otherwise, dirstate are fully restored by 'dirstateguard' For example in timetable above, timestamp of "f" in .hg/dirstate is restored to -1 (or less than N), and subsequent 'hg status' can detect changes correctly. Case (2) always causes 'repo.status()' invocation via 'repo.commit()' just after changing files inside same wlock scope. ---- ----------------------------------- ---------------- timestamp of "f" ---------------- dirstate file- time action mem file system ---- ----------------------------------- ---- ----- ----- N *** *** - make file "f" clean N - execute 'hg foobar' .... - 'dirstate.normal("f")' N *** (e.g. via dirty check or previous 'repo.commit()') - change "f", but keep size N - 'repo.status()' (*1) (via 'repo.commit()') ---- ----------------------------------- ---- ----- ----- At a glance, 'repo.status()' at (*1) seems to cause similar issue (= "changed files are treated as clean"), but actually doesn't. 'dirstate._lastnormaltime' should be N at (*1) above, because 'dirstate.normal()' via dirty check is finished at N. Therefore, "f" changed at N (= 'dirstate._lastnormaltime') is forcibly treated as "unsure" at (*1), and changes are detected as expected (see 'dirstate.status()' for detail). If 'hg import' is executed with '--no-commit', 'repo.status()' isn't invoked just after changing files inside same wlock scope. But preceding 'dirstate.normal()' is invoked inside another wlock scope via 'cmdutil.bailifchanged()', and in-memory changes should be flushed at the end of that scope. Therefore, timestamp N of clean "f" should be replaced by -1, if 'dirstate.write()' is invoked at N. It means that condition of this issue isn't satisfied.

tests: add extension to emulate invoking internalpatch at the specific time This extension fakes "mtime" of patched files to emulate invoking 'patch.internalpatch()' at the specific time. This is useful to reproduce timing critical issues fixed in subsequent patches.

cmdutil: remove useless dirstate.normallookup() invocation in revert() Explicit 'dirstate.normallookup()' invocation in 'revert()' is useless now, because previous patch fixed the relevant issue by writing in-memory dirstate changes out at the end of dirty check. 'dirstate.normallookup()' invocation was introduced by 21b33f0460e0 to avoid occasional test failure (see issue4583 for detail). This is partial backout of it (added tests are still left).

merge: remove useless dirstate.normallookup() invocation in applyupdates() Explicit 'dirstate.normallookup()' invocation via 'dirtysubstate()' in 'applyupdates()' is useless now, because previous patch fixed the relevant issue by writing in-memory dirstate changes out at the end of dirty check. 'dirstate.normallookup()' invocation was introduced by 6becb9dbca25 to avoid occasional test failure. This is partial backout of it (added tests are still left).

context: write dirstate out explicitly after marking files as clean To detect change of a file without redundant comparison of file content, dirstate recognizes a file as certainly clean, if: (1) it is already known as "normal", (2) dirstate entry for it has valid (= not "-1") timestamp, and (3) mode, size and timestamp of it on the filesystem are as same as ones expected in dirstate This works as expected in many cases, but doesn't in the corner case that changing a file keeps mode, size and timestamp of it on the filesystem. The timetable below shows steps in one of typical such situations: ---- ----------------------------------- ---------------- timestamp of "f" ---------------- dirstate file- time action mem file system ---- ----------------------------------- ---- ----- ----- N -1 *** - make file "f" clean N - execute 'hg foobar' - instantiate 'dirstate' -1 -1 - 'dirstate.normal("f")' N -1 (e.g. via dirty check) - change "f", but keep size N N+1 - release wlock - 'dirstate.write()' N N - 'hg status' shows "f" as "clean" N N N ---- ----------------------------------- ---- ----- ----- The most important point is that 'dirstate.write()' is executed at N+1 or later. This causes writing dirstate timestamp N of "f" out successfully. If it is executed at N, 'parsers.pack_dirstate()' replaces timestamp N with "-1" before actual writing dirstate out. Occasional test failure for unexpected file status is typical example of this corner case. Batch execution with small working directory is finished in no time, and rarely satisfies condition (2) above. This issue can occur in cases below; - 'hg revert --rev REV' for revisions other than the parent - failure of 'merge.update()' before 'merge.recordupdates()' The root cause of this issue is that files are changed without flushing in-memory dirstate changes via 'repo.commit()' (even though omitting 'dirstate.normallookup()' on changed files also causes this issue). To detect changes of files correctly, this patch writes in-memory dirstate changes out explicitly after marking files as clean in 'workingctx._checklookup()', which is invoked via 'repo.status()'. After this change, timetable is changed as below: ---- ----------------------------------- ---------------- timestamp of "f" ---------------- dirstate file- time action mem file system ---- ----------------------------------- ---- ----- ----- N -1 *** - make file "f" clean N - execute 'hg foobar' - instantiate 'dirstate' -1 -1 - 'dirstate.normal("f")' N -1 (e.g. via dirty check) ----------------------------------- ---- ----- ----- - 'dirsttate.write()' -1 -1 ----------------------------------- ---- ----- ----- - change "f", but keep size N N+1 - release wlock - 'dirstate.write()' -1 -1 - 'hg status' -1 -1 N ---- ----------------------------------- ---- ----- ----- To reproduce this issue in tests certainly, this patch emulates some timing critical actions as below: - timestamp of "f" in '.hg/dirstate' is -1 at the beginning 'hg debugrebuildstate' before command invocation ensures it. - make file "f" clean at N - change "f" at N 'touch -t 200001010000' before and after command invocation changes mtime of "f" to "2000-01-01 00:00" (= N). - invoke 'dirstate.write()' via 'repo.status()' at N 'fakedirstatewritetime.py' forces 'pack_dirstate()' to use "2000-01-01 00:00" as "now", only if 'pack_dirstate()' is invoked via 'workingctx._checklookup()'. - invoke 'dirstate.write()' via releasing wlock at N+1 (or "not at N") 'pack_dirstate()' via releasing wlock uses actual timestamp at runtime as "now", and it should be different from the "2000-01-01 00:00" of "f". BTW, this patch also changes 'test-largefiles-misc.t', because adding 'dirstate.write()' makes recent dirstate changes visible to external process.

tests: add extension to emulate invoking dirstate.write at the specific time This extension fakes 'now' for 'parsers.pack_dirstate()' to emulate invoking 'dirstate.write()' at the specific time, only when 'dirstate.write()' is invoked via functions below: - 'workingctx._checklookup()' (= 'repo.status()') - 'committablectx.markcommitted()' This is useful to reproduce timing critical issues fixed in subsequent patches.

convert: handle copies when converting from Perforce (issue4744)

convert: add config for recording the source name This creates the convert.hg.sourcename config option which will embed a user defined name into each commit created by the convert. This is useful when using the convert extension to merge several repositories together and we want to record where each commit came from.

convert: support multiple specifed revs in git source This allows specifying multiple revs/branches to convert from a git repo.

convert: add support for specifying multiple revs Previously convert could only take one '--rev'. This change allows the user to specify multiple --rev entries. For instance, this could allow converting multiple branches (but not all branches) at once from git. In this first patch, we disable support for this for all sources. Future patches will enable it for select sources (like git).

monoblue: use padding instead of position for text in footer Some installations alter monoblue style and remove margins from body element (these margins have that dark gray background) to adapt hgweb instance to an already existing site design. However, the margins hid a quirk in page footer: a block of text needlessly popped out of the footer, and when margins were gone, the whole page got a vertical scroll bar because of that. Live example: https://hg.prosody.im/prosody-modules/ To remove the potential scroll bar, this block of text now uses left padding, which doesn't make it overflow the footer, but makes it achieve the otherwise same result visually.

monoblue: don't try to show repo on hgwebdir index page Index page shows a list of accessible repositories, it doesn't have a single-repo context.

convert: handle deleted files when converting from Perforce (issue4743)

bookmarks: change bookmark within a transaction For some time, bookmark can and should be moved in the transaction. This changeset migrates the 'hg bookmarks' commands to use a transaction. Tests regarding rollback and transaction hooks are impacted for obvious reasons. Some have to be slightly updated to keep testing the same things. Some can just be dropped because they do not make sense anymore.

bookmark: remove the "touch changelog" hack Any changes to bookmarks used to touch the changelog to ensure hgweb was reloaded. This was fairly hacky and stops working when bookmarks are moved as part of the transaction. As hgweb is now explicitly tracking bookmark changes, we can remove this hack (and no tests break).

convert: add config option for disabling ancestor parent checks When converting merge commits, convert checks if any of the parents are ancestors of any of the other parents. To do this, it builds an ancestor list for every commit in the repository. On large repos this can take a long time (30min+). Let's add an option for disabling this check to preserve performance. The downside of this is that it may create unnecessary parent connections when enabled (which is unfortunate, but not incorrect). To verify, I ran the convert tests with the flag enabled, and verified the graph changes were all just to add new parents that were ancestors of existing parents.

convert: add config to not convert tags In some cases we do not want to convert tags from the source repo to be tags in the target repo (for instance, in a large repository, hgtags cause scaling issues so we want to avoid them). This adds a config option to disable converting tags.

templatekw: make {rev} return wdirrev instead of None wdirrev/wdirnode identifiers are still experimental, but {node} is mapped to wdirnode. So {rev} should do the same for consistency. I'm not sure if templatekw can import scmutil. If not, we should move intrev() to node module.

changeset_printer: use node.wdirrev to calculate meaningful parentrevs Because we defined the working-directory revision is INT_MAX, it makes sense that "hg log -r 'wdir()'" displays the "parent:" field. This is the same for two revisions that are semantically contiguous but the intermediate revisions are hidden.

workingctx: use node.wdirid constant

node: define experimental identifiers for working directory The "ff..." node was introduced at 183965a00c76, and we also need an integer that can be processed in revset. We could use len(repo), but it would be likely to hide possible bugs. Instead, using INT_MAX, we can notice such bugs by IndexError, at the cost of handling non-contiguous revisions.

templatekw: apply manifest template only if ctx.manifestnode() exists This will prevent crash by "hg log -r 'wdir()' -Tdefault". We could use the pseudo ff... hash introduced by 183965a00c76, but it isn't proven idea yet. For now, I want to make "hg log" just works in order to test 'wdir()' revset. Note that unlike its name, "{manifest}" is not a list of files in that revision, but a pair of (manifestrev, manifestnode).

merge with stable

import-checker: exclude mercurial packages installed into the system path If mercurial was installed into a directory other than the site-packages, test-module-imports.t failed as 'mercurial.node' was listed in stdlib_modules: testpackage/latesymbolimport.py relative import of stdlib module Instead, we should exclude our packages explicitly.

import-checker: recurse into subtree of sys.path only if __init__.py exists We can't assume that the site-packages is the only directory that has Python files but is not handled as a package. For example, we have dist-packages directory on Debian.

hghave: allow adding customized features at runtime Before this patch, there is no way to add customized features to `hghave` without changing `hghave` and `hghave.py` themselves. This decreases reusability of `run-tests.py` framework for third party tools, because they may want to examine custom features at runtime (e.g. existence of some external tools). To allow adding customized features at runtime, this patch makes `hghave` import `hghaveaddon` module, only when `hghaveaddon.py` file can be found in directories below: - `TESTDIR` for invocation via `run-tests.py` - `.` for invocation via command line The path to the directory where `hghaveaddon.py` should be placed is added to `sys.path` only while importing `hghaveaddon`, because: - `.` may not be added to `PYTHONPATH` - adding additional path to `sys.path` may change behavior of subsequent `import` for other features `hghave` is terminated with exit code '2' at failure of `import hghaveaddon`, because exit code '2' terminates `run-tests.py` immediately. This is a one of preparations for issue4677.

import-checker.py: exit with code 0 if no error is detected Before this patch, `import-checker.py` exits with non-0 code, if no error is detected. This is unusual as Unix command. This change may be a one of preparations for issue4677, because this can avoid extra explanation about unusual exit code of `import-checker.py` for third party tool developers.

run-tests.py: add TESTDIR to PATH if it differs from RUNTESTDIR Before this patch, `RUNTESTDIR` is added to `PATH`, but `TESTDIR` isn't. This doesn't cause any problems, if `run-tests.py` runs in `tests` directory of Mercurial source tree. In this case, `RUNTESTDIR` should be equal to `TESTDIR`. On the other hand, if `run-tests.py` runs in `tests` of third party tools, commands in that directory should be executed with explicit `$TESTDIR/` prefix in `*.t` test scripts. This isn't suitable for the policy "drop explicit $TESTDIR from executables" of Mercurial itself (see 4d2b9b304ad0). BTW, 4d2b9b304ad0 describes that "$TESTDIR is added to the path" even though `TESTDIR` isn't added to `PATH` exactly speaking, because `TESTDIR` and `RUNTESTDIR` weren't yet distinguished from each other at that time. This is a one of preparations for issue4677.

run-tests.py: add RUNTESTDIR to refer `tests` of Mercurial Before this patch, there is no way to refer files under `tests` or so of Mercurial source tree, when `run-tests.py` runs in `tests` of third party tools. In this case, `TESTDIR` refers the latter `tests`. This prevents third party tools from using useful tools in Mercurial source tree (e.g. `contrib/check-code.py`). This patch adds `RUNTESTDIR` environment variable to refer `tests` of Mercurial source tree, in which `run-tests.py` now running is placed. For example, tests of third party tools can refer `contrib/check-code.py` in Mercurial source tree as `$RUNTESTDIR/../contrib/check-code.py`. BTW, for similarity with `TESTDIR` referring `test*s*` directory, newly added environment variable isn't named as `RUNTEST*S*DIR`. In addition to it, the corresponded local variable is also named as `runtestdir`. This is a one of preparations for issue4677.

run-tests.py: execute hghave by the path relative to run-tests.py Before this patch, `run-tests.py` executes `hghave` by the path relative to `TESTDIR` (= cwd of `run-tests.py` running). This prevents third party tools for Mercurial from running `run-tests.py`, which is placed in `tests` of Mercurial source tree, in `tests` of own source tree. In such cases, `TESTDIR` refers the latter `tests`, and `hghave` doesn't exist in it. This is a one of preparations for issue4677.

templatekw: make {latesttag} a hybrid list This maintains the previous behavior of expanding {latesttag} to a string containing all of the tags, joined by ':'. But now it also allows list type operations. I'm unsure if the plural handling is correct (i.e. it seems like it is usually "{foos % '{foo}'}"), but I guess we are stuck with this because the singular form previously existed.

templatekw: allow the caller of showlist() to specify the join() separator The keyword {latesttag} currently manually joins the list of tags using ':', which prevents a transparent switch over to a hybrid list.

archive: use {changessincelatesttag} to build the metadata file This isolates the current magic when dealing with wdir() and only().

templatekw: introduce the changessincelatesttag keyword Archive is putting a value with the same name in the metadata file, to count all of the changes not covered by the latest tag, instead of just along the longest path. It seems that this would be useful to have on the command line as well. It might be nice for the name to start with 'latesttag' so that it is grouped with the other tag keywords, but I can't think of a better name. The initial version of this counted a clean wdir() and '.' as the same value, and a dirty wdir() as the same value after it is committed. Yuya objected on the grounds of consistency [1]. Since revsets can be used to conditionally select a dirty wdir() or '.' when clean, I can build the version string I need and will defer to him on this. [1] https://www.selenic.com/pipermail/mercurial-devel/2015-June/071588.html

help: support 'hg help template.somekeyword' Previously the output was simply 'abort: help section not found'.

test-convert-git: use a relative gitmodule url The absolute URL was causing this error with 1.9.5 on Windows, which had a cascading effect: @@ -466,22 +466,24 @@ > url = $TESTTMP/git-repo5 > EOF $ git commit -a -m "weird white space submodule" - [master *] weird white space submodule (glob) - Author: nottest <test@example.org> - 1 file changed, 3 insertions(+) + fatal: bad config file line 6 in $TESTTMP/git-repo6/.gitmodules + [128] $ cd .. $ hg convert git-repo6 hg-repo6 initializing destination hg-repo6 repository scanning source... For reasons unknown, there is still this delta on Windows: @@ -490,7 +490,6 @@ $ git commit -q -m "missing .gitmodules" $ cd .. $ hg convert git-repo6 hg-repo6 --traceback - fatal: Path '.gitmodules' does not exist in '*' (glob) initializing destination hg-repo6 repository scanning source... sorting...

test-convert-git: stablize for git 1.7.7.6 The output has apparently changed slightly since this version. Since they are just commits without any obvious importance to the test, and I can't figure out how to glob them away, silence them. Sample diffs were like this: @@ -468,7 +468,7 @@ $ git commit -a -m "weird white space submodule" [master *] weird white space submodule (glob) Author: nottest <test@example.org> - 1 file changed, 3 insertions(+) + 1 files changed, 3 insertions(+), 0 deletions(-) $ cd .. $ hg convert git-repo6 hg-repo6 initializing destination hg-repo6 repository

hgweb: also monitor change to bookmarks This makes changes to bookmarks visible to hgweb through the official way. There is no change to tests because there is currently another hack in place to ensure the same behavior.

hgweb: also refresh the repo on changes to the obsstore Before this change, hgweb could miss update to the obsolescence markers store if that was the only change between two commands.

hgweb: use an extensible list of files to check for refresh The refresh feature was explicitly testing if '00changelog.i' and 'phaseroots' changed. This is overlooking other important information like bookmarks and obsstore (bookmark have their own hack to work around it). We move to a more extensible system with a list of files of interest that will be used to build the repo state. The system should probably move into a more central place so that the command server and other systems are able to use it. Extension writers will also be able to add entries to ensure that changes to extension data are properly detected. Also the current key (mtime, size) is notably weak for bookmarks and phases whose files can easily change content without effect on their size. Still, this patch seems like a valuable minimal first step.

hgweb: drop the default argument for get_stat This default argument is used twice and is making things confusing. Making it explicit helps to clarify coming changesets

revset: prefetch method in "parents" As already demonstrated, saving attribute lookup gains us some minor but noticeable performance improvements. revset #0: parents(all()) before) 0.024169 after ) 0.022756 94%

convert: handle deleted files when converting from Perforce (issue4743)

forget: add a note to the command help about remove

shelve: only keep the latest N shelve backups This will keep the backup directory from growing indefinitely. The number of backups to keep can be set using the shelve.maxbackups config option (defaults to 10 backups).

shelve: always backup shelves instead of deleting them Instead of being deleted, shelve files are now moved into the .hg/shelve-backup directory. This is designed similarly to how strip saves backups into .ht/strip-backup. The goal is to prevent data loss especially when using unshelve. There are cases in which a user can complete an unshelve but lose some of the data that was shelved by, for example, resolving merge conflicts incorrectly. Storing backups will allow the user to recover the data that was shelved, at the expense of using more disk space over time.

wireproto: remove a debug print This looks like someone forgot something here.