tests: use the system hg for examining the local repository Most test scripts use "hg" to interact with a temporary test repository. However a few tests also want to run hg commands to interact with the local repository containing the mercurial source code. Notably, many of the test-check-* tests want to check local files and commit messages. These tests were previously using the version of hg being tested to query the source repository. However, this will fail if the source repository requires extensions or other settings not supported by the version of mercurial being tested. The source repository was typically initially cloned using the system hg installation, so we should use the system hg installation to query it. There was already a helpers-testrepo.sh script designed to help cope with different requirements for the source repository versus the test repositories. However, it only handled the evolve extension. This new behavior works with any extensions that are different between the system installation and the test installation.

tests: save the original PATH and PYTHONPATH variables When running the tests, define ORIG_PATH and ORIG_PYTHONPATH environment variables that contain the original contents of PATH and PYTHONPATH, before they were modified by run-tests.py This will make it possible for tests to refer to the original contents of these variables if necessary. In particular, this is necessary for invoking the correct version of hg for examining the local repository (the mercurial repository itself, not the temporary test repositories). Various tests examine the local repository to check the file lists and contents of commit messages.

setup: prefer using the system hg to interact with the local repository Add a findhg() function that tries to be smarter about figuring out how to run hg for examining the local repository. It first tries running "hg" from the user's PATH, with the default HGRCPATH settings intact, but with HGPLAIN enabled. This will generally use the same version of mercurial and the same settings used to originally clone the repository, and should have a higher chance of working successfully than trying to run the hg script from the local repository. If that fails findhg() falls back to the existing behavior of running the local hg script.

setup: replace runhg() with an hgcommand helper class Replace the runhg() function with an hgcommand helper class. hgcommand has as run() function similar to runhg(), but no longer requires the caller to pass in the exact path to python and the hg script, and the environment settings for invoking hg. For now this diff contains no behavior changes, but in the future this will make it easier for the hgcommand helper class to more intelligently figure out the proper way to invoke hg.

setup: move environment computation into a helper function Add a helper function to compute the environment used for invoking mercurial, rather than doing this computation entirely at global scope. This will make it easier to do some subsequent refactoring.

setup: update runcmd() to also return the exit status Update the runcmd() helper function so it also returns the process exit status. This allows callers to more definitively determine if a command failed, rather than testing only for the presence of data on stderr. I don't expect this to have any behavioral changes for now: the commands invoked by setup generally should print data on stderr if and only if they failed.

setup: fail if we cannot determine the version number If running hg fails, exit the setup script unsuccessfully, rather than proceeding to use a bogus version of "+0-". Using an invalid version number causes various tests to fail later. Failing early makes it easier to identify the source of the problem. It is currently easy for setup.py to fail this way since it sets HGRCPTH to the empty string before running "hg", which may often disable extensions necessary to interact with the local repository.

smartset: fix generatorset.last() to not return the first element (issue5609)

strip: respect the backup option in stripcallback The backup option was mistakenly ignored. It should be respected. Thanks Martin von Zweigbergk for finding this out!

py3: use pycompat.bytestr() to convert str to bytes

py3: pass the memoryview object into bytes() to get the value

py3: use pycompat.bytestr instead of str

py3: use '%d' to convert integers to bytes

py3: slice over bytes to prevent getting it's ascii value

py3: use pycompat.strkwargs() to convert kwargs keys to str

py3: use r'' to prevent the addition of b'' by transformer There are cases in opts handling in debugcommands.py where we don't need to convert opts keys back to bytes as there are some handful cases and no other function using opts value. Using r'', we prevent the transformer to add a b'' which will keep the value str.

py3: use pycompat.byteskwargs() to convert kwargs' keys to bytes This is used where ever required like where kwargs are passed into ui.formatter(), scmutil.match() or cmdutil.openrevlog() which expects bytes.

tests: use cgienv to minimize environment setup at hgweb tests This patch follows other hgweb tests.

tests: avoid test failure for mangling path-like string by MSYS

py3: make sure commands name are bytes in tests

py3: add b'' to make the regex pattern bytes

py3: use hex() to convert the hash to bytes

py3: add b'' to make a triple quoted string bytes on Python 3 Transformer does not adds b'' in front of triple quoted strings to prevent converting docs to bytes.

py3: add tests to show `hg bookmarks` and `hg branches` work on Python 3

py3: fix kwargs handling for `hg bookmarks`

identify: provide changectx to templater

formatter: proxy fm.context() through converter Otherwise nested template formatter would not see the context objects. It's just a boolean flag now. We might want to change it to 'ctxs -> items' function so changectx attributes are populated automatically in JSON, but I'm not sure.

identify: change p1/p2 to a list of parents It makes sense because the nested data structure is a list of items.

scmutil: add a cleanupnodes method for developers It's now common that an old node gets replaced by zero or more new nodes, that could happen with amend, rebase, histedit, etc. And it's a common requirement to do bookmark movements, strip or obsolete nodes and even moving working copy parent. Previously, amend, rebase, history have their own logic doing the above. This patch is an attempt to unify them and future code. This enables new developers to be able to do "replace X with Y" thing correctly, without any knowledge about bookmarks, strip or obsstore. The next step will be migrating rebase to the new API, so it works inside a transaction, and its code could be simplified.

strip: add a delayedstrip method that works in a transaction For long, the fact that strip does not work inside a transaction and some code has to work with both obsstore and fallback to strip lead to duplicated code like: with repo.transaction(): .... if obsstore: obsstore.createmarkers(...) if not obsstore: repair.strip(...) Things get more complex when you want to call something which may call strip under the hood. Like you cannot simply write: with repo.transaction(): .... rebasemod.rebase(...) # may call "strip", so this doesn't work But you do want rebase to run inside a same transaction if possible, so the code may look like: with repo.transaction(): .... if obsstore: rebasemod.rebase(...) obsstore.createmarkers(...) if not obsstore: rebasemod.rebase(...) repair.strip(...) That's ugly and error-prone. Ideally it's possible to just write: with repo.transaction(): rebasemod.rebase(...) saferemovenodes(...) This patch is the first step towards that. It adds a "delayedstrip" method to repair.py which maintains a postclose callback in the transaction object.

workingfilectx: add audit() as a wrapper for wvfs.audit()

workingfilectx: add backgroundclose as a kwarg to write() This is necessary because some callers in merge.py pass backgroundclose=True when writing. As with previous changes in this series, this should be a no-op.

merge: change repo.wvfs.setflags calls to a new wctx[f].setflags function As with previous changes in this series, this should be a no-op.

merge: convert repo.wwrite() calls to wctx[f].write() As with the previous patch in this series, workingfilectx.write() is a direct call to repo.wwrite(), so this change should be a no-op.

merge: replace repo.wvfs.unlinkpath() with calls to wctx[f].remove() The code inside workingfilectx.remove() is a straight call to repo.wvfs.unlinkpath, so this should be a no-op.

merge: pass wctx to batchremove and batchget We would like to migrate direct calls of repo.wvfs/wwrite/wread/etc to a call on the relevant workingfilectx, both as a cleanup (to reduce the number of working copy functions on `repo`), and also to facilitate an in-memory merge that doesn't write to the working copy. In order to do that, the first step is to ensure we pass the target wctx around and perform our writes and reads on it. Later, this object might become a memctx.

revset: add depth limit to descendants() (issue5374) This is naive implementation using two-pass scanning. Tracking descendants isn't an easy problem if both start and stop depths are specified. It's impractical to remember all possible depths of each node while scanning from roots to descendants because the number of depths explodes. Instead, we could cache (min, max) depths as a good approximation and track ancestors back when needed, but that's likely to have off-by-one bug. Since this implementation appears not significantly slower, and is quite straightforward, I think it's good enough for practical use cases. The time and space complexity is O(n) ish. revisions: 0) 1-pass scanning with (min, max)-depth cache (worst-case quadratic) 1) 2-pass scanning (this version) repository: mozilla-central # descendants(0) (for reference) *) 0.430353 # descendants(0, depth=1000) 0) 0.264889 1) 0.398289 # descendants(limit(tip:0, 1, offset=10000), depth=1000) 0) 0.025478 1) 0.029099 # descendants(0, depth=2000, startdepth=1000) 0) painfully slow (due to quadratic backtracking of ancestors) 1) 1.531138

dagop: make walk direction switchable so it can track descendants # ancestors(tip) using hg repo 2) 0.068527 3) 0.069097

dagop: factor out generator of ancestor nodes # ancestors(tip) using hg repo 1) 0.068976 2) 0.068527

dagop: factor out pfunc from revancestors() generator This generator will be reused for tracking descendants with depth limit. # ancestors(tip) using hg repo 0) 0.065868 1) 0.068976

dagop: use smartset.min() in revdescendants() generator All callers pass the result of revset.getset(), which should be a smartset.

dagop: change revdescendants() to include all root revisions Prepares for adding depth support. I want to process depth=0 in revdescendants() to make things simpler. only() also calls dagop.revdescendants(), but it filters out root revisions explicitly. So this should cause no problem. # descendants(0) using hg repo 0) 0.052380 1) 0.051226 # only(tip) using hg repo 0) 0.001433 1) 0.001425

test-revset: add a few more tests of descendants() I'll add depth support to descendants().

dagop: unnest inner generator of revdescendants() This just moves iterate() to module-level function.

smartset: fix default value of abstractsmartset.sort() It's unused, but it shouldn't lie.

keyword: wrap functions only once at loading keyword extension Before this patch, some functions are wrapped in reposetup(), but this causes redundant nested wrapping, if two ore more repositories enable keyword extension (e.g. hgweb serves multiple repositories). Now, there is no need to define these wrapper functions in reposetup(), because previous patches made them not directly refer to kwtemplater instanciated in reposetup(). This patch factors these wrapper functions out from reposetup(), and uses them to wrap functions only at once at loading keyword extension.

keyword: use _keywordkwt of repository instead of kwtools['templater'] Now, kwtemplater instance can be obtained via _keywordkwt property of repository.

keyword: obtain kwtemplater instance via repository at runtime Wrapper functions of keyword extension are defined in reposetup(), because they refer to kwtemplater instantiated in reposetup(). This patch makes them obtain kwtemplater instance via repository at runtime. For reviewability, this patch focuses on wrapper functions, which handle generator. This is a part of preparations for defining them statically.