tests: add extension to emulate invoking dirstate.write at the specific time This extension fakes 'now' for 'parsers.pack_dirstate()' to emulate invoking 'dirstate.write()' at the specific time, only when 'dirstate.write()' is invoked via functions below: - 'workingctx._checklookup()' (= 'repo.status()') - 'committablectx.markcommitted()' This is useful to reproduce timing critical issues fixed in subsequent patches.

convert: handle copies when converting from Perforce (issue4744)

convert: add config for recording the source name This creates the convert.hg.sourcename config option which will embed a user defined name into each commit created by the convert. This is useful when using the convert extension to merge several repositories together and we want to record where each commit came from.

convert: support multiple specifed revs in git source This allows specifying multiple revs/branches to convert from a git repo.

convert: add support for specifying multiple revs Previously convert could only take one '--rev'. This change allows the user to specify multiple --rev entries. For instance, this could allow converting multiple branches (but not all branches) at once from git. In this first patch, we disable support for this for all sources. Future patches will enable it for select sources (like git).

monoblue: use padding instead of position for text in footer Some installations alter monoblue style and remove margins from body element (these margins have that dark gray background) to adapt hgweb instance to an already existing site design. However, the margins hid a quirk in page footer: a block of text needlessly popped out of the footer, and when margins were gone, the whole page got a vertical scroll bar because of that. Live example: https://hg.prosody.im/prosody-modules/ To remove the potential scroll bar, this block of text now uses left padding, which doesn't make it overflow the footer, but makes it achieve the otherwise same result visually.

monoblue: don't try to show repo on hgwebdir index page Index page shows a list of accessible repositories, it doesn't have a single-repo context.

convert: handle deleted files when converting from Perforce (issue4743)

bookmarks: change bookmark within a transaction For some time, bookmark can and should be moved in the transaction. This changeset migrates the 'hg bookmarks' commands to use a transaction. Tests regarding rollback and transaction hooks are impacted for obvious reasons. Some have to be slightly updated to keep testing the same things. Some can just be dropped because they do not make sense anymore.

bookmark: remove the "touch changelog" hack Any changes to bookmarks used to touch the changelog to ensure hgweb was reloaded. This was fairly hacky and stops working when bookmarks are moved as part of the transaction. As hgweb is now explicitly tracking bookmark changes, we can remove this hack (and no tests break).

convert: add config option for disabling ancestor parent checks When converting merge commits, convert checks if any of the parents are ancestors of any of the other parents. To do this, it builds an ancestor list for every commit in the repository. On large repos this can take a long time (30min+). Let's add an option for disabling this check to preserve performance. The downside of this is that it may create unnecessary parent connections when enabled (which is unfortunate, but not incorrect). To verify, I ran the convert tests with the flag enabled, and verified the graph changes were all just to add new parents that were ancestors of existing parents.

convert: add config to not convert tags In some cases we do not want to convert tags from the source repo to be tags in the target repo (for instance, in a large repository, hgtags cause scaling issues so we want to avoid them). This adds a config option to disable converting tags.

templatekw: make {rev} return wdirrev instead of None wdirrev/wdirnode identifiers are still experimental, but {node} is mapped to wdirnode. So {rev} should do the same for consistency. I'm not sure if templatekw can import scmutil. If not, we should move intrev() to node module.

changeset_printer: use node.wdirrev to calculate meaningful parentrevs Because we defined the working-directory revision is INT_MAX, it makes sense that "hg log -r 'wdir()'" displays the "parent:" field. This is the same for two revisions that are semantically contiguous but the intermediate revisions are hidden.

workingctx: use node.wdirid constant

node: define experimental identifiers for working directory The "ff..." node was introduced at 183965a00c76, and we also need an integer that can be processed in revset. We could use len(repo), but it would be likely to hide possible bugs. Instead, using INT_MAX, we can notice such bugs by IndexError, at the cost of handling non-contiguous revisions.

templatekw: apply manifest template only if ctx.manifestnode() exists This will prevent crash by "hg log -r 'wdir()' -Tdefault". We could use the pseudo ff... hash introduced by 183965a00c76, but it isn't proven idea yet. For now, I want to make "hg log" just works in order to test 'wdir()' revset. Note that unlike its name, "{manifest}" is not a list of files in that revision, but a pair of (manifestrev, manifestnode).

merge with stable

import-checker: exclude mercurial packages installed into the system path If mercurial was installed into a directory other than the site-packages, test-module-imports.t failed as 'mercurial.node' was listed in stdlib_modules: testpackage/latesymbolimport.py relative import of stdlib module Instead, we should exclude our packages explicitly.

import-checker: recurse into subtree of sys.path only if __init__.py exists We can't assume that the site-packages is the only directory that has Python files but is not handled as a package. For example, we have dist-packages directory on Debian.

hghave: allow adding customized features at runtime Before this patch, there is no way to add customized features to `hghave` without changing `hghave` and `hghave.py` themselves. This decreases reusability of `run-tests.py` framework for third party tools, because they may want to examine custom features at runtime (e.g. existence of some external tools). To allow adding customized features at runtime, this patch makes `hghave` import `hghaveaddon` module, only when `hghaveaddon.py` file can be found in directories below: - `TESTDIR` for invocation via `run-tests.py` - `.` for invocation via command line The path to the directory where `hghaveaddon.py` should be placed is added to `sys.path` only while importing `hghaveaddon`, because: - `.` may not be added to `PYTHONPATH` - adding additional path to `sys.path` may change behavior of subsequent `import` for other features `hghave` is terminated with exit code '2' at failure of `import hghaveaddon`, because exit code '2' terminates `run-tests.py` immediately. This is a one of preparations for issue4677.

import-checker.py: exit with code 0 if no error is detected Before this patch, `import-checker.py` exits with non-0 code, if no error is detected. This is unusual as Unix command. This change may be a one of preparations for issue4677, because this can avoid extra explanation about unusual exit code of `import-checker.py` for third party tool developers.

run-tests.py: add TESTDIR to PATH if it differs from RUNTESTDIR Before this patch, `RUNTESTDIR` is added to `PATH`, but `TESTDIR` isn't. This doesn't cause any problems, if `run-tests.py` runs in `tests` directory of Mercurial source tree. In this case, `RUNTESTDIR` should be equal to `TESTDIR`. On the other hand, if `run-tests.py` runs in `tests` of third party tools, commands in that directory should be executed with explicit `$TESTDIR/` prefix in `*.t` test scripts. This isn't suitable for the policy "drop explicit $TESTDIR from executables" of Mercurial itself (see 4d2b9b304ad0). BTW, 4d2b9b304ad0 describes that "$TESTDIR is added to the path" even though `TESTDIR` isn't added to `PATH` exactly speaking, because `TESTDIR` and `RUNTESTDIR` weren't yet distinguished from each other at that time. This is a one of preparations for issue4677.

run-tests.py: add RUNTESTDIR to refer `tests` of Mercurial Before this patch, there is no way to refer files under `tests` or so of Mercurial source tree, when `run-tests.py` runs in `tests` of third party tools. In this case, `TESTDIR` refers the latter `tests`. This prevents third party tools from using useful tools in Mercurial source tree (e.g. `contrib/check-code.py`). This patch adds `RUNTESTDIR` environment variable to refer `tests` of Mercurial source tree, in which `run-tests.py` now running is placed. For example, tests of third party tools can refer `contrib/check-code.py` in Mercurial source tree as `$RUNTESTDIR/../contrib/check-code.py`. BTW, for similarity with `TESTDIR` referring `test*s*` directory, newly added environment variable isn't named as `RUNTEST*S*DIR`. In addition to it, the corresponded local variable is also named as `runtestdir`. This is a one of preparations for issue4677.

run-tests.py: execute hghave by the path relative to run-tests.py Before this patch, `run-tests.py` executes `hghave` by the path relative to `TESTDIR` (= cwd of `run-tests.py` running). This prevents third party tools for Mercurial from running `run-tests.py`, which is placed in `tests` of Mercurial source tree, in `tests` of own source tree. In such cases, `TESTDIR` refers the latter `tests`, and `hghave` doesn't exist in it. This is a one of preparations for issue4677.

templatekw: make {latesttag} a hybrid list This maintains the previous behavior of expanding {latesttag} to a string containing all of the tags, joined by ':'. But now it also allows list type operations. I'm unsure if the plural handling is correct (i.e. it seems like it is usually "{foos % '{foo}'}"), but I guess we are stuck with this because the singular form previously existed.

templatekw: allow the caller of showlist() to specify the join() separator The keyword {latesttag} currently manually joins the list of tags using ':', which prevents a transparent switch over to a hybrid list.

archive: use {changessincelatesttag} to build the metadata file This isolates the current magic when dealing with wdir() and only().

templatekw: introduce the changessincelatesttag keyword Archive is putting a value with the same name in the metadata file, to count all of the changes not covered by the latest tag, instead of just along the longest path. It seems that this would be useful to have on the command line as well. It might be nice for the name to start with 'latesttag' so that it is grouped with the other tag keywords, but I can't think of a better name. The initial version of this counted a clean wdir() and '.' as the same value, and a dirty wdir() as the same value after it is committed. Yuya objected on the grounds of consistency [1]. Since revsets can be used to conditionally select a dirty wdir() or '.' when clean, I can build the version string I need and will defer to him on this. [1] https://www.selenic.com/pipermail/mercurial-devel/2015-June/071588.html

help: support 'hg help template.somekeyword' Previously the output was simply 'abort: help section not found'.

test-convert-git: use a relative gitmodule url The absolute URL was causing this error with 1.9.5 on Windows, which had a cascading effect: @@ -466,22 +466,24 @@ > url = $TESTTMP/git-repo5 > EOF $ git commit -a -m "weird white space submodule" - [master *] weird white space submodule (glob) - Author: nottest <test@example.org> - 1 file changed, 3 insertions(+) + fatal: bad config file line 6 in $TESTTMP/git-repo6/.gitmodules + [128] $ cd .. $ hg convert git-repo6 hg-repo6 initializing destination hg-repo6 repository scanning source... For reasons unknown, there is still this delta on Windows: @@ -490,7 +490,6 @@ $ git commit -q -m "missing .gitmodules" $ cd .. $ hg convert git-repo6 hg-repo6 --traceback - fatal: Path '.gitmodules' does not exist in '*' (glob) initializing destination hg-repo6 repository scanning source... sorting...

test-convert-git: stablize for git 1.7.7.6 The output has apparently changed slightly since this version. Since they are just commits without any obvious importance to the test, and I can't figure out how to glob them away, silence them. Sample diffs were like this: @@ -468,7 +468,7 @@ $ git commit -a -m "weird white space submodule" [master *] weird white space submodule (glob) Author: nottest <test@example.org> - 1 file changed, 3 insertions(+) + 1 files changed, 3 insertions(+), 0 deletions(-) $ cd .. $ hg convert git-repo6 hg-repo6 initializing destination hg-repo6 repository

hgweb: also monitor change to bookmarks This makes changes to bookmarks visible to hgweb through the official way. There is no change to tests because there is currently another hack in place to ensure the same behavior.

hgweb: also refresh the repo on changes to the obsstore Before this change, hgweb could miss update to the obsolescence markers store if that was the only change between two commands.

hgweb: use an extensible list of files to check for refresh The refresh feature was explicitly testing if '00changelog.i' and 'phaseroots' changed. This is overlooking other important information like bookmarks and obsstore (bookmark have their own hack to work around it). We move to a more extensible system with a list of files of interest that will be used to build the repo state. The system should probably move into a more central place so that the command server and other systems are able to use it. Extension writers will also be able to add entries to ensure that changes to extension data are properly detected. Also the current key (mtime, size) is notably weak for bookmarks and phases whose files can easily change content without effect on their size. Still, this patch seems like a valuable minimal first step.

hgweb: drop the default argument for get_stat This default argument is used twice and is making things confusing. Making it explicit helps to clarify coming changesets

revset: prefetch method in "parents" As already demonstrated, saving attribute lookup gains us some minor but noticeable performance improvements. revset #0: parents(all()) before) 0.024169 after ) 0.022756 94%

convert: handle deleted files when converting from Perforce (issue4743)

forget: add a note to the command help about remove

shelve: only keep the latest N shelve backups This will keep the backup directory from growing indefinitely. The number of backups to keep can be set using the shelve.maxbackups config option (defaults to 10 backups).

shelve: always backup shelves instead of deleting them Instead of being deleted, shelve files are now moved into the .hg/shelve-backup directory. This is designed similarly to how strip saves backups into .ht/strip-backup. The goal is to prevent data loss especially when using unshelve. There are cases in which a user can complete an unshelve but lose some of the data that was shelved by, for example, resolving merge conflicts incorrectly. Storing backups will allow the user to recover the data that was shelved, at the expense of using more disk space over time.

wireproto: remove a debug print This looks like someone forgot something here.

amend: move obsmarkers creation in the "new changeset" conditional We already check if we created a new changesets right above this piece of code, so we can just drop the condition and indent the markers creation.

amend: move createmarkers evaluation earlier The value is used at multiple points in the function. Retrieving the value in the middle of the transaction scope gives the false impression that it has a single user. We move it at the start of the function to clarify this.

wireproto: correctly escape batched args and responses (issue4739) This issue appears to be as old as wireproto batching itself: I can reproduce the failure as far back as 08ef6b5f3715 trivially by rebasing the test changes in this patch, which was back in the 1.9 era. I didn't test before that change, because prior to that the testfile has a different name and I'm lazy. Note that the test thought it was checking this case, but it actually wasn't: it put a literal ; in the arg and response for its greet command, but the mangle/unmangle step defined in the test meant that instead of "Fo, =;o" going over the wire, "Gp-!><p" went instead, which doesn't contain any special characters (those being [.=;]) and thus not exercising the escaping. The test has been updated to use pre-unmangled special characters, so the request is now "Fo+<:o", which mangles to "Gp,=;p". I have confirmed that the test fails without the adjustment to the escaping rules in wireproto.py. No existing clients of RPC batching were depending on the old behavior in any way. The only *actual* users of batchable RPCs in core were: 1) largefiles, wherein it batches up many statlfile calls. It sends hexlified hashes over the wire and gets a 0, 1, or 2 back as a response. No risk of special characters. 2) setdiscovery, which was using heads() and known(), both of which communicate via hexlified nodes. Again, no risk of special characters. Since the escaping functionality has been completely broken since it was introduced, we know that it has no users. As such, we can change the escaping mechanism without having to worry about backwards compatibility issues. For the curious, this was detected by chance: it happens that the lz4-compressed text of a test file for remotefilelog compressed to something containing a ;, which then caused the failure when I moved remotefilelog to using batching for file content fetching.

merge with stable

revset: port extra() to support keyword arguments This is an example to show how keyword arguments are processed.

revset: add function to build dict of positional and keyword arguments Keyword arguments will be convenient for functions that will take more than one optional or boolean flags. For example, file(pattern[, subrepos=false]) subrepo([[pattern], status]) Because I don't think all functions should accept key=value syntax, getkwargs() does not support variadic functions such as 'ancestor(*changeset)'. The core logic is placed in the parser module because keyword arguments will be more useful in the templater, where functions take more options. Test cases will be added by the next patch.

revset: add parsing rule for key=value pair It will be used as an keyword argument. Note that our "=" operator is left-associative. In general, the assignment operator is right-associative, but we don't care because it isn't allowed to chain "=" operations.

import-checker: establish modern import convention We introduce a new convention for declaring imports and enforce it via the import checker script. The new convention is only active when absolute imports are used, which is currently nowhere. Keying off "from __future__ import absolute_import" to engage the new import convention seems like the easiest solution. It is also beneficial for Mercurial to use this mode because it means less work and ambiguity for the importer and potentially better performance due to fewer stat() system calls because the importer won't look for modules in relative paths unless explicitly asked. Once all files are converted to use absolute import, we can refactor this code to again only have a single import convention and we can require use of absolute import in the style checker. The rules for the new convention are documented in the docstring of the added function. Tests have been added to test-module-imports.t. Some tests are sensitive to newlines and source column position, which makes docstring testing difficult and/or impossible.

import-checker: establish new function for verifying import conventions A future patch will formalize the modern import convention. In preparation for that, introduce a new wrapper function that will invoke the proper function.

import-checker: resolve relative imports "from . import X" will produce an ImportFrom ast node with .module = None. This resulted in a run-time error from attempting to concatenate None with a str. Another problem with relative imports is that the prefix may be dynamic based on the "level" attribute of the import. e.g. "from ." has level 1 and "from .." has level 2. We teach the "fromlocal" function how to cope with relative imports. Where appropriate, the consumer passes in the level so relative module names may be resolved properly.

templatekw: use a list of tags in getlatesttags() instead of joining them This will be used in the next patch. It also points out that the documentation for '{latesttag}' is not quite accurate, since it says "most recent global tag" (singular). I assume it is too radical of a change to convert it to a list of strings. At least ':' is currently a reserved character in tag names.

convert: improve support for unusual .gitmodules Previously convert would throw an exception if it encountered a git commit with a .gitmodules file that was malformed (i.e. was missing, but had submodule files, or was malformed). Instead of breaking the convert entirely, let's print error messages and move on.

convert: handle .gitmodules with non-tab whitespaces The old implementation assumed .gitmodules file lines always began with tabs. It can be any whitespace, so lets trim the lines appropriately.

convert: fix bug with converting the same commit twice Convert had a bug where it relied on repo.tip() to be the newly committed commit. This was not the case if the commit already existed in the repository (since repo.commitctx() did nothing, the tip() referenced some random other commit and the revmap got corrupted). This fixes it by using the node returned by repo.commitctx().

templater: remove workaround for escaped quoted string in quoted template This patch backs out 554d6fcc3c84 which should no longer be needed. The test for '{\"invalid\"}' is removed because the parser is permissive for \"...\" literal.

merge with stable

amend: stop updating the bookmarks twice There was code to move the bookmarks around both in the 'cmdutil' help and in the main 'commit' function. We kill the 'commit' version as it is performed outside the transaction. The debug note is moved into cmdutil.

amend: collaborate with the transaction when moving bookmarks We have code moving bookmarks from the old changeset to the new one within the transaction scope. Yet this code was still writing to disk instead of handing the change to the transaction. This changeset fixes this.