tests: fix up some lax escaping in test-template-basic.t These misfired escapes turn into hard errors in Python 3.7, and I'd really rather we not work around it. We should *probably* try and find a way to proactively warn users about invalid escape sequences. There's one more failure of this type in this file on Python 3.7, but I can't figure out the issue. It'll need to be corrected in a follow-up. Differential Revision: https://phab.mercurial-scm.org/D3843

cleanup: migrate from re.escape to stringutil.reescape This has consistent behavior on Python 2.7, 3.6, and 3.7 and has the benefit of probably being a little faster. Test output changes are largely because / used to be pointlessly escaped. Differential Revision: https://phab.mercurial-scm.org/D3842

stringutil: add a new function to do minimal regex escaping Per https://bugs.python.org/issue29995, re.escape() used to over-escape regular expression strings, but in Python 3.7 that's been fixed, which also improved the performance of re.escape(). Since it's both an output change for us *and* a perfomance win, let's just effectively backport the new behavior to hg on all Python versions. Differential Revision: https://phab.mercurial-scm.org/D3841

graft: add no-commit mode (issue5631) This patch adds a new flag --no-commit in graft command. This feature grafts the changes but do not create commits for those changes, grafted changes will be added in the working directory. Also added tests to reflect the expected behavior. Differential Revision: https://phab.mercurial-scm.org/D2409

patchbomb: use email.mime.base instead of email.MIMEBase The later was removed on Python 3. On python 2: >>> email.MIMEBase.MIMEBase is email.mime.base.MIMEBase True Differential Revision: https://phab.mercurial-scm.org/D3836

patchbomb: use email.mime.multipart instead of email.MIMEMultipart The later was removed in Python 3. On python 2: >>> email.MIMEMultipart.MIMEMultipart is email.mime.multipart.MIMEMultipart True Differential Revision: https://phab.mercurial-scm.org/D3835

py3: add b'' prefixes in tests/test-bundle2-pushback.t This makes the test run on Python 3. # skip-blame because just b'' prefixes. Differential Revision: https://phab.mercurial-scm.org/D3834

py3: make tests/test-diff-antipatience.t work with python 3 On python3, if we do list(b'abcd'), we get [97, 98, 99, 100]. So this patch does the list() operation on unicodes and then convert things to bytes. This also adds `and None` to suppress return values of .write() calls. Differential Revision: https://phab.mercurial-scm.org/D3833

terse: pass "clean" and "unknown" booleans by name for clarity Differential Revision: https://phab.mercurial-scm.org/D3838

terse: add tests of running from subdirectory I also included comments explaining what I think the output should be (discussion started on D3628). Differential Revision: https://phab.mercurial-scm.org/D3837

convert: don't pass --no-files to "darcs show repo" command The test fails with darcs 2.14.0 because --no-files is no longer supported. It was removed at the patch 97973a52bf496657558412562d6fad2ee651b1e0, which says "show repo: removed --files option, removed manual flags parsing." As far as I can tell, --no-files was specified just for optimization, so old darcs should work without it.

shelve: stop testing missing rebase state file We no longer use the file. Neither during rebase not during continue. Differential Revision: https://phab.mercurial-scm.org/D3691

shelve: actually test corrupted shelve state The test was previous moving the corrupted shelve state file as an histedit state file. Differential Revision: https://phab.mercurial-scm.org/D3690

shelve: directly handle `--continue` Shelve is currently sub-contracting some of its work to the rebase extension. In order to make shelve more independent and flexible we would like shelve to handle the parent alignment directly. This changeset takes on the next step, handling the abort process. Same as for --abort. It turns out we have all the necessary bits in the `shelvestate` file. So we do not need anything from the interrupted rebase. Differential Revision: https://phab.mercurial-scm.org/D3689

shelve: directly handle the abort process Shelve is currently sub-contracting some of its work to the rebase extension. In order to make shelve more independent and flexible we would like shelve to handle the parent alignment directly. This change starts with the simplest bits, handling the abort process. It turns out we have all the necessary bits in the `shelvestate` file. So we do not need anything from the interrupted rebase. As a nice side effect, a test about missing `shelverebasestate` state file now behave better. Differential Revision: https://phab.mercurial-scm.org/D3688

shelve: check the actual shelvestate in morestatus The rebasestate is a lower level implementation details that we are trying to remove. Differential Revision: https://phab.mercurial-scm.org/D3687

packaging: dynamically define make targets We currently have make boilerplate for each instance of a distro's release. This is redundant, annoying to maintain, and prone to errors. This commit defines variables holding available releases for various distros. We then iterate through the list and dynamically define make targets. Differential Revision: https://phab.mercurial-scm.org/D3761

packaging: don't write files for templatized Dockerfiles Now that Docker image building is implemented in Python and we can perform template substitution in memory, we don't need to write out produced Dockerfiles to disk. Differential Revision: https://phab.mercurial-scm.org/D3760

packaging: replace dockerlib.sh with a Python script I want to do some more advanced things with Docker in upcoming commits. Trying to do that with shell scripts will be a bit too painful for my liking. Implementing things in Python will be vastly simpler in the long run. This commit essentially ports dockerlib.sh to a Python script. dockerdeb and dockerrpm have been ported to use the new hg-docker script. hg-docker requires Python 3. I've only tested on Python 3.5. Unlike the local packaging scripts which may need to run on old distros, the Docker packaging scripts don't have these constraints. So I think it is acceptable to require Python 3.5. As part of the transition, the Docker image tags changed slightly. I don't think that's a big deal: the Docker image names are effectively arbitrary and are a means to an end to achieve running commands in Docker containers. The code for resolving the Dockerfile content allows substituting values passed as arguments. This will be used in a subsequent commit. Differential Revision: https://phab.mercurial-scm.org/D3759

packaging: consistently create build user in Dockerfiles Previously, dockerlib.sh appended some commands to create a "build" user in each Docker image. The resulting Docker images could be inconsistent depending on the execution environment and base image. With this change, we explicitly create our custom user and group as the first action in each Dockerfile. The user always has user:group 1000:1000 and all built images are consistent. We also create a home directory for the user under /build. This directory is currently ignored. As part of this, we stop setting the DBUILDUSER variable in dockerlib.sh and instead set it in the respective scripts that call it. This is in preparation for further refactoring of dockerlib.sh. Differential Revision: https://phab.mercurial-scm.org/D3758

scmutil: move construction of instability count message to separate fn When the commad we are running, introduces new instabilities, we show a message like `5 new orphan changesets`, `2 new content-divergent changesets`, `1 new phase-divergent changesets` etc which is very nice. Now taking a step ahead, we want users to show how to fix them too. Something like: `5 new orphan changesets (run 'hg evolve' to resolve/stabilize them)` `2 new content-divergent changesets (run 'hg evolve --content-divergent' to resolve them)` and maybe telling user a way to understand more about those new instabilities like `hg evolve --list` or `hg log -r 'orphan()'` something like that. The idea came from issue5855 which I want to fix because fixing that will result in a nice UI. Taking the construction logic out will allow extensions like evolve (maybe rebase too) to wrap that and add information about how to resolve and how to understand the instability more. Differential Revision: https://phab.mercurial-scm.org/D3734

procutil: use unbuffered stdout on Windows Windows doesn't support line buffering, treating it as fully buffered. This causes output of slow commands to stutter. We use unbuffered instead.

procutil: use unbuffered stdout on Windows Windows doesn't support line buffering, treating it as fully buffered. This causes output of slow commands to stutter. We use unbuffered instead.

graft: introduce --abort flag to abort interrupted graft This patch introduces a new --abort flag to `hg graft` command which aborts an interrupted graft and rollbacks to the state before graft. The behavior when some of grafted changeset get's published while interrupted graft or we have new descendants on grafted changesets is same as that of rebase which is warn the user, don't strip and abort the abort the graft. Tests are added for the new flag. .. feature:: `hg graft` now has a `--abort` flag which aborts the interrupted graft and rollbacks to state before the graft. Differential Revision: https://phab.mercurial-scm.org/D3754

graft: move `if continue` to elif and add new line This will make upcoming patch where we introduce a new elif for the abort case more readable. Also added a new line before the if-else starts. Differential Revision: https://phab.mercurial-scm.org/D3752

graft: start storing new nodes formed in graftstate This patch starts storing the new nodes formed during the ongoing graft operation in the graftstate. We need the list of new nodes formed while implmenting `graft --abort` which will strip out the new nodes. Differential Revision: https://phab.mercurial-scm.org/D3751

show: use filter() function to strip "tip" tag Before, an empty tag "" was inserted in place of "tip", resulting in double spaces.

templater: extend filter() to accept template expression for emptiness test This utilizes the pass-by-name nature of template arguments.

templater: introduce filter() function to remove empty items from list The primary use case is to filter out "tip" from a list of tags.

templater: fix truth testing of integer 0 taken from a list/dict Broken at f9c426385853. bool(python_value) shouldn't be used here since an integer 0 has to be truthy for backward compatibility.

formatter: look for template symbols from the associated name Otherwise symbolsused() would fail if a named template is specified with -T.

py3: add b'' prefixes in tests/test-obsolete-divergent.t This makes the test pass on Python 3. # skip-blame because just b'' prefixes. Differential Revision: https://phab.mercurial-scm.org/D3832

py3: use stringutil.pprint() to print NoneType Before this patch, when running test-debugcommands.t, we get a TypeError because NoneType can't be converted into bytes. This patch uses stringutil.pprint() to print the ui._colormode. We are now close to getting test-debugcommands.t passing on Python 3. Differential Revision: https://phab.mercurial-scm.org/D3831

rebase: make dry-run return 1 or 0 according to result In dry-run mode, if there is no conflict return 0, if any then return 1 Differential Revision: https://phab.mercurial-scm.org/D3829

relink: use context manager for lock management

censor: use context manager for lock management

templater: remove redundant member variables from templater class We no longer need them since the engine is instantiated in __init__().

templater: resurrect cache of engine instance The engine-level cache was effectively disabled at 48289eafb37d "templater: drop extension point of engine classes (API)" by mistake, which made template rendering quite slow. Spotted by Martin von Zweigbergk.

templater: extract template loader to separate class This avoids reference cycle in the subsequent patch: templater -> _proc -> templater.load -> templater The templater class will be a thin wrapper around the loader and the engine.

tests: rename and document test-command-template.t So that we can specify templater tests as test-template-*. (original) $ wc -l test-command-template.t test-template-* 5041 test-command-template.t 23 test-template-filters.t 5064 total (at this patch) $ wc -l test-template-* 1088 test-template-basic.t 1376 test-template-functions.t 1195 test-template-keywords.t 1760 test-template-map.t 5419 total

tests: extract test-template-keywords.t from test-command-template.t

tests: fold test-template-filters.t into test-template-functions.t

tests: extract test-template-functions.t from test-command-template.t I decided to not split filters and functions into two test files since we sometimes reimplement a filter as a function.

tests: extract test-template-map.t from test-command-template.t test-command-template.t is one of the slowest tests. Let's split it into 4 files of manageable size.

cleanupnodes: preserve phase of parents of new nodes As Yuya noted in the review of D3818, passing in targetphase=phases.draft would result in advancing the phase boundary of a secret-phase parent. We never pass targetphase=phases.draft so far, but it's a bug waiting to happen. I tried to refactor it so max(parentphase, X) happened in one place, but I couldn't come up with good variables names and I ended up with a "newphase = max(newphase, parentphase)" line, which made the whole block not look any better to me. Differential Revision: https://phab.mercurial-scm.org/D3824

rebase: delete the comment which was not following "do not eat my data" Differential Revision: https://phab.mercurial-scm.org/D3756

tags: unblock log-like template keywords and functions It checks if ctx will be used in template since loading ctx per revision could take extra 10-100msec in total depending on the number of tags.

formatter: provide hint of context keys required by template This allows us to create ctx objects only if necessary. I tried another idea which is to populate ctx from 'repo' and 'node' value on demand. It worked, but seemed unnecessarily complicated. So I chose a simpler one. The datafields argument is a space-separated string for consistency with fm.write() API.

templatefuncs: declare resource requirements for future use

templatefuncs: minimize resource requirements

merge with stable

merge: do not fill manifest of committed revision with pseudo node (issue5526) Since a75d24539aba "convert: fix convert dropping p2 contents during filemap merge", wctx is not always a committablectx because the convert extension passes in repo[n] as wctx. If wctx is a committed changeset, its manifest dict shouldn't be mutated reflecting to the working directory.

split: preserve phase of commit that is being split With this change, hg split will preserve the phase of the commit that is being split, ignoring the phases.new-commit setting. Previously, we would use whatever phases.new-commit was set to (unless our parent was secret, then we would be secret even if phases.new-commit=draft). Now, splitting a draft commit with phases.new-commit=secret does not cause the new commits to become secret, and splitting a secret commit with phases.new-commit=draft and a draft parent does not cause the new commits to become draft. Test cases and commit message taken from Kyle Lippincott's D2016 (thanks!). Differential Revision: https://phab.mercurial-scm.org/D3819

scmutil: make cleanupnodes optionally also fix the phase We have had multiple bugs where the phase wasn't correctly carried forward to a rewritten changeset (for example: phabricator, split, evolve, fix). Handling the phase update in cleanupnodes() makes it less likely to happen again, especially once we have made it fix the phase by default (perhaps in the next release cycle). This patch also updates all applicable callers so we get some testing of it. Note that rebase and histedit can't be fixed yet because they call cleanupnodes() only at the end and the phase may have been changed by the user when the rebase/histedit was interrupted (due to merge conflicts). I think we should make them write one commit at a time (as it already does), along with associated obsmarkers, bookmark moves, etc. When that's done, we can switch them over to cleanupnodes(). Differential Revision: https://phab.mercurial-scm.org/D3818

tests: add test of uncommit with default phase as secret We didn't seem to have any test checking that uncommitting a draft commit with phase.new-phase=secret preserved the draft phase. Differential Revision: https://phab.mercurial-scm.org/D3817

grep: add --diff flag Adds a diff flag, which works exactly same as all, in fact since --all searches diffs, there diff is a better name for it. The all flag is still here for backward compatibility reasons. Some major tests for all has been picked and added for diff. Differential Revision: https://phab.mercurial-scm.org/D3763

fix: include cleanupnodes() in transaction As pointed out by Yuya, we need a transaction to make sure the state before the call to cleanupnodes() is not observable. Differential Revision: https://phab.mercurial-scm.org/D3823

progress: enforce use of complete() on the helper class complete() is preferred over update(None), so let's enforce that. Differential Revision: https://phab.mercurial-scm.org/D3822

progress: extract function for closing topic progress(None) had a completely different implementation from the progress(<not None>) implementation. It very much feels like it should be a separate method, so this patch makes it so. That also makes it clear that only the topic parameter matters when closing a topic (e.g. "total" does not matter). Differential Revision: https://phab.mercurial-scm.org/D3821

progress: use context manager for lock Differential Revision: https://phab.mercurial-scm.org/D3820