fix: add suboption for configuring execution order of tools This could be accomplished by using wrapper scripts, but that would diminish the usefulness of the incremental formatting logic. Configuring execution order along with other things in the hgrc is probably more convenient anyway. This change highlights some awkwardness with suboptions and default values, which should be addressed separately. Differential Revision: https://phab.mercurial-scm.org/D5237

tests: fix up some import statements caught by Python 3 I'm curious how the import checker manages to be so much more pedantic in Python 3, but not enough to bother exploring. Differential Revision: https://phab.mercurial-scm.org/D5240

tests: replace `tac` reimplementation by `sort -r` `sort -r` is better code "formatter" than `tac` since it's stable. It's also portable so we don't need to reimplement it in Python. Differential Revision: https://phab.mercurial-scm.org/D5239

tests: work around `tac` not being portable Introduce a tac.py helper and use it. Sigh. Differential Revision: https://phab.mercurial-scm.org/D5238

revlog: give formatting to clang-format This has two lines that are poorly formatted, both right after PyObject_HEAD macros. It's possible that in the future there will be a feature to tell clang-format that a token includes an end-of-line character (in this case the ;), but for now I'm willing to live with two poorly-formatted lines in a struct definition in exchange for not thinking about how this file is formatted. Differential Revision: https://phab.mercurial-scm.org/D5234

revlog: add blank line in comment to help clang-format Differential Revision: https://phab.mercurial-scm.org/D5233

revlog: add a comment to help clang-format produce less-awful results Differential Revision: https://phab.mercurial-scm.org/D5232

py3: ratchet caught two more passing tests Differential Revision: https://phab.mercurial-scm.org/D5236

perf: add a lazydeltabase option to perfrevlogwrite The option controls if we'll test the provided delta first. This is the current default behavior for unbundle.

perf: add `storage` as possible source for perfrevlogwrite This source will use the stored delta.

perf: add `parent-smallest` as possible source for perfrevlogwrite This source will use the smallest of the possible diff against parent.

perf: add `parent-2` as possible source for perfrevlogwrite This source will use a diff against p2 if it exists and fall back to p1 otherwise.

perf: add `parent-1` as possible source for perfrevlogwrite This source will use a diff against p1 in all case.

perf: add the notion of "source" to perfrevlogwrite We want to test performance associated witch various way to add a new revision. They will be specified using this new argument.

perf: only display the total time for perfrevlogwrite if quiet This provide a simple way to get an overview of the total performance.

perf: offer full details in perfrevlogwrite This will be useful for people who want to study the timing pattern more closely.

perf: introduce a perfrevlogwrite command The command record times taken by adding many revisions to a revlog. Timing each addition, individually. The "added revision" are recreations of the original ones. To time each addition individually, we have to handle the timing and the reporting ourselves. This command is introduced to track the impact of sparse-revlog format on delta computations at initial storage time. It starts with the full text, a situation similar to the "commit". Additions from an existing delta are better timed with bundles. The complaints from `check-perf-code.py` are not relevant. We are accessing and "revlog" opener, not a repository opener.

tests: fix config knob in test-narrow-clone-stream.t Two patches landed in parallel and had a semantic conflict. This resolves the mess and leaves us with passing tests. Differential Revision: https://phab.mercurial-scm.org/D5231

remotefilelog: fix various whitespace issues in docstring Differential Revision: https://phab.mercurial-scm.org/D5230

ui: add config knob to redirect status messages to stderr (API) This option can be used to isolate structured output from status messages. For now, "stdio" (stdout/err pair) and "stderr" are supported. In future patches, I'll add the "channel" option which will send status messages to a separate command-server channel with some metadata attached, maybe in CBOR encoding. This is a part of the generic templating plan: https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Sanity_check_output .. api:: Status messages may be sent to a dedicated stream depending on configuration. Don't use ``ui.status()``, etc. as a shorthand for conditional writes. Use ``ui.write()`` for data output.

phabricator: ensure the command summaries are available in extension help Previously, `hg help phabricator` listed the 3 supported commands at the bottom of the extension help, but said "no help text available".

hgweb: cast bytearray to bytes PEP-3333 seems to indicate that bytes is the only allowed type that can be used to express the output of a WSGI application. And some WSGI environments seem to enforce this (mod_wsgi does). This commit universally casts bytearray instances to bytes to appease the WSGI specification. I found this because wireprotov2 is emitting bytearray instances. I'd like to keep things that way because the way it builds a data structure, bytearray is more efficient. I'd rather keep the low-level code efficient (and using bytearray) and cast at the edges than impose a performance penalty on code that may run outside WSGI contexts.

help: unjumble the list of default config values for `internals.config`

ui: hide fin/fout/ferr attributes behind @property functions This allows keeping references to fout/ferr/fin which are updated when these properties are changed. See the next patch.

ui: label prompt and echo messages I'm going to add a dedicated command-server channel for status messages, which carries metadata alongside a message text. 'ui.*' label provides a hint how message text should be processed.

ui: add inner function to select write destination I'm going to add a config knob to redirect any status messages to stderr. This function helps to switch underlying file objects. # no-check-commit because of existing write_err() function

ui: remove _write() and _write_err() functions

ui: move pre/post processes from low-level write()s to _writenobuf() This helps adding a dedicated stream for status/error messages. I don't want to add _write*() function per stream.

ui: pass in file object to _writenobuf() See the subsequent patches for why. The "if" block in _writenobuf() will be removed soon.

perf: fix perfrevlogrevisions --reverse Currently, 'endrev' equals `len(revlog)`, a revision that does not exist. When asking for the reverse order, the arguments passed to xrange are `xrange(len(revlog), startrev)` which then crash. We need to offset 'endrev' by one so we don't crash anymore. Also, we offset 'startrev' to ensure we get the same number of revisions with and without the `--reverse` option. Differential Revision: https://phab.mercurial-scm.org/D5228

procutil: import concerns about creationflags on Windows from D1701 I don't have the need anymore for the change in D1701 nor the time to investigate the changes on all supported Windows platforms. I import the stuff I learned on D1701 in the `runbgcommand` so the next people working on it can starts from there. Differential Revision: https://phab.mercurial-scm.org/D5229

localrepo: extract loading of hgrc files to standalone function Various 3rd party extensions supplement where per-repo config data lives. Looking at their sources, they resort to unorthodox means to inject the config data. And the way they do it is susceptible to corner cases. e.g. not processing automatic extension loads, not reacting to new or disabled extensions in configs, etc. This commit extracts the core logic of loading hgrc files into a standalone function so there is a clear function that can be monkeypatched to inject per-repo config data at repository open time. Differential Revision: https://phab.mercurial-scm.org/D5221

revsets: make bookmark/named('re:nonexistent') not abort (issue6018) (BC) Foozy documented the differences between revsets branch(), tag(), bookmark(), and named() in eeb5d5ab14a6 (revset: raise RepoLookupError to make present() predicate continue the query, 2015-01-31). He seemed to want tag() to change behavior to not error out on non-matching regular expressions. I think it's instead bookmark() and named() that should not error out. So that's what this patch does. Differential Revision: https://phab.mercurial-scm.org/D5220

fix: rename :fileset subconfig to :pattern This name was always inaccurate, since the config accepts any pattern. Hopefully so few people use this right now that it won't matter, but there will now be a warning if the old config name is used. Differential Revision: https://phab.mercurial-scm.org/D5226

fix: add a config to abort when a fixer tool fails This allows users to stop and address tool failures before proceeding, instead of the default behavior of continuing to apply any tools that didn't fail. For example, a code formatting tool could fail if you have syntax errors, and you might want your repo to stay in its current state while you fix the syntax error before re-running 'hg fix'. It's conceivable that this would even be necessary for the correctness of some fixer tools across a chain of revisions. Differential Revision: https://phab.mercurial-scm.org/D5200

perf: measure slicing time in perfrevlogrevision Slicing a sparse delta chain can be expensive. We now benchmark the associated time.

perf: teach perfrevlogrevision about sparse reading Before this change, chunks were always read in a single block. Even in the sparse-read/sparse-revlog case. This gave a false view of the performance and could lead to memory consumption issue.

perf: use the same timer for all section of perfrevlogrevision Otherwise the -T json output is invalid.

obsutil: clarify the access to "repo" We use the variable multiple times and we might use it even more in the future. We use a temporary variable instead.

pycompat: adding Linux detection and fixing Mac Python 3 recommends detecting OSs with the prefix of the platform, but we were comparing the full string for macOS. We also didn't have Linux detection, which is convenient for extensions to use (rather than have some OSs detected by hg and some by the extension). Reference: https://docs.python.org/3/library/sys.html#sys.platform Differential Revision: https://phab.mercurial-scm.org/D5227

remotefilelog: fix various minor py3 problems # skip-blame b prefixes and pycompat.long, nothing remotely interesting Differential Revision: https://phab.mercurial-scm.org/D5223

remotefilelog: rip out a missed mention of lz4 Differential Revision: https://phab.mercurial-scm.org/D5222

ui: wrap whole _write() block with timeblockedsection I think the cost of color labeling is negligible compared to the I/O syscalls. Let's simply wrap the whole write() function so that we can eliminate _write() and _write_err() in later changeset.

ui: indent _writenobuf() to prepare moving bits from _write() functions

ui: simply concatenate messages before applying color labels This should be cheaper in space than applying labels for each message.

ui: simplify interface of low-level write() functions _write() and _write_err() will be replaced with fout.write() and ferr.write() respectively. This is the first step.

ui: factor out function that writes data to fout/ferr with labeling I'm thinking of adding an option to send status messages to stderr (or a dedicated command-server channel) so that structured output (e.g. JSON) would never be interleaved with non-formatter output. A unified write() interface helps to do that.

ui: consolidate places where _progclear() is called The progress bar has to be cleared when we start writing some data to the output stream. Let's make it always triggered immediately before switching by _colormode, so that we can easily factor out helper functions.

templater: compute revset lazily This speeds up e.g. "{ifcontains(rev, revset('::.'), ...)}" in common cases where 'rev' is near the working parent. The templater API is ugly, but it helps here. 'f' can be either a generator or a function returning a generator.

remotefilelog: drop compat code for "getbundle_shallow" wireprotocol command Doing some annotate on hgexperimental shows that getbundle_shallow used to exist in 2013 or before. We don't have any pre-2013 remotefilelog users except Fb themselves and I doubt they are going to use in-core remotefilelog. So it's safe to remove this. Differential Revision: https://phab.mercurial-scm.org/D5193

tests: remove lz4 as dependency while running tests One of the previous patch authored by Augie rips out the lz4 dependency and things should work without it. Now there are just 2-3 tests failing because of same change in emitrevisions() API. Differential Revision: https://phab.mercurial-scm.org/D5192

remotefilelogserver: add a matcher argument to _walkstreamfiles() Implementing narrow stream clones in core, I added an optional matcher argument. The function in remotefilelogserver.py does not know about that argument and does not accept that and hence some tests fails. Differential Revision: https://phab.mercurial-scm.org/D5191

shallowutil: introduce a helper function isenabled() This patch introduces a inenabled() function which will check whether remotefilelog is enabled or not. The function is then also used at all the places where check whether remotefilelog is enabled or not. The new function makes code easy to read without need to understand what is the constant involved and why we are checking repo.requirements. Differential Revision: https://phab.mercurial-scm.org/D5190

remotefilelog: remove some BC code related to streamclone Since remotefilelog is now moved to core and we have streamclone._walkstreamfiles() in core, we don't need to have the backward compatibility code. People with old mercurial version should use remotefilelog from hg-experimental repo as IMO remotefilelog will go under a good refactoring and old clients will break. # no-check-commit foo_bar function name Differential Revision: https://phab.mercurial-scm.org/D5189

remotefilelog: add some docstring Differential Revision: https://phab.mercurial-scm.org/D5131

remotefilelog: rename wireproto methods and remaining capabilities This is a mess, in part because there should be more constants throughout. I know we typically do exp- instead of the x_ business in this change, but I also had to use this in some function names, so I figured until I can break that coupling I'd go with this. If it's too unpleasant during review, let me know and I can probably clean it up some more. # no-check-commit due to new foo_bar naming - too hard to avoid right now :( Differential Revision: https://phab.mercurial-scm.org/D5129

remotefilelog: consolidate and rename on-disk store requirement The value of this constant appeared in too many places. While we're here, rename it to be more consistent with our naming conventions for experimental functionality. Differential Revision: https://phab.mercurial-scm.org/D5128

remotefilelog: consolidate and rename bundle2 capability Bonus: we catch a spot where the shallowrepo requirement was being misused as the bundle2 capability. Differential Revision: https://phab.mercurial-scm.org/D5127

remotefilelog: rename capability for legacy ssh file fetching method I think I want to delete this, but for now we'll just rename this so it's easy to tell apart from other strings that have the same value. Differential Revision: https://phab.mercurial-scm.org/D5126

remotefilelog: rip out lz4 support Some methods are left teased out so it's easier to build a migration extension from v1 packs to v2. I also anticipate those methods will be of use if we get around to adding configurable compression engines before we end up jettisoning packs. I think it would make sense to register one-shot `compress` and `decompress` methods on our compression engines in util.py, but indygreg mentioned that might not be straightforward. Differential Revision: https://phab.mercurial-scm.org/D4944

remotefilelog: bump pack file formats to version 2 Drop support for format 0 and 1 packs. Version 2 will be different in that it'll use zlib compression instead of lz4, as the latter isn't really suitable for core. If I can make it work, I'll make compression engines pluggable, but the logic for handling these files is spread out enough it's a little tricky to get that right. Test changes are only due to pack files being renamed. The contents of the packs are the same. Differential Revision: https://phab.mercurial-scm.org/D4943

remotefilelog: avoid hard-coding pack names in tests I'm about to alter the pack format, which alters the content-addressed name of the pack file. In order to reduce my own confusion I found it helpful to use an ls | head pipeline since we always want to inspect the newest pack. Differential Revision: https://phab.mercurial-scm.org/D4942

remotefilelog: remove now-unused wirepack code Differential Revision: https://phab.mercurial-scm.org/D5124

remotefilelogserver: remove pack-serving functionality Differential Revision: https://phab.mercurial-scm.org/D5123

remotefilelog: remove support for fetching packs We've never used this at Google, and it's strictly an optimization that indygreg and I are confident we can handle more elegantly later. Let's rip it out to keep the surface area of the imported remotefilelog small. Some tests have changes because the tests for regular packs and repacking were mixed with tests of fetching packs. I'm pretty sure I've done the right thing here. Differential Revision: https://phab.mercurial-scm.org/D5122

remotefilelog: jettison example cache client We can always reconstruct it from history if it becomes interesting. Differential Revision: https://phab.mercurial-scm.org/D4941

remotefilelog: prune obsolete method Differential Revision: https://phab.mercurial-scm.org/D4940

logtoprocess: use new runbgcommand from procutil Differential Revision: https://phab.mercurial-scm.org/D4939

procutil: port over windows encoding fixes from logtoprocess Differential Revision: https://phab.mercurial-scm.org/D5121

remotefilelog: transplant runbgcommand to procutil While cleaning up the deprecated runshellcommand I noticed a near-clone of this in logtoprocess, so I'm standardizing on what appears to be the newer one by moving it to procutil. Differential Revision: https://phab.mercurial-scm.org/D4938

remotefilelog: remove function that was described as deprecated Differential Revision: https://phab.mercurial-scm.org/D4937

remotefilelog: import pruned-down remotefilelog extension from hg-experimental This is remotefilelog as of my recent patches for compatibility with current tip of hg, minus support for old versions of Mercurial and some FB-specific features like their treemanifest extension and fetching linkrev data from a patched phabricator. The file extutil.py moved from hgext3rd to remotefilelog. This is not yet ready to be landed, consider it a preview for now. Planned changes include: * replace lz4 with zstd * rename some capabilities, requirements and wireproto commands to mark them as experimental * consolidate bits of shallowutil with related functions (eg readfile) I'm certainly open to other (small) changes, but my rough mission is to land this largely as-is so we can use it as a model of the functionality we need going forward for lazy-fetching of file contents from a server. # no-check-commit because of a few foo_bar functions Differential Revision: https://phab.mercurial-scm.org/D4782

store: introduce _matchtrackedpath() and use it to filter store files This patch introduces a function to filter store files on the basis of the path which they are tracking. The function assumes that the entries can be of two types, 'meta/*' and 'data/*' which means it will just work on revlog based storage and not with another storage ways. For the 'data/*' entries, we remove the 'data/' part and '.i/.d' part from the beginning and the end then pass that to matcher. For the 'meta/*' entries, we remove the 'meta/' and '/00manifest.(i/d)' part from beginning and end then call matcher.visitdir() with it to make sure all the parent directories are also downloaded. Since the storage filtering for narrow stream clones is implemented with this patch, we remove the un-implemented error message, add some more tests and add the treemanifest case to tests too. The tests demonstrate that it works correctly. After this patch, we have now narrow stream clones working. Narrow stream clones are a very important feature for large repositories who have good internet connection because they use streamclones for cloning and if they do normal narrow clone, that takes more time then a full streamclone. Also narrow-stream clone will drastically speed up clone timings. Differential Revision: https://phab.mercurial-scm.org/D5139

narrow: only send includepats and excludepats if they are not empty If we send an empty includepats or excludepats argument to getbundle, it's translated to `['']` on the server which causes problems because even though it's empty, bool of that value if True and we end up creating differencematcher with narrowspec.match() which results in unexpected behavior. Differential Revision: https://phab.mercurial-scm.org/D5138

exchange: pass includepats and excludepats as arguments to getbundle() This will help in implementing narrow stream clones. Also narrow extension used to add these arguments, now we add them by default if they are not empty. Since reading includepats and excludepats on the server only works when narrow is enabled, we check if narrow if enabled or not before passing them. Differential Revision: https://phab.mercurial-scm.org/D5119

catapult: add a bit more documentation on how to use catapult tracing Differential Revision: https://phab.mercurial-scm.org/D5217

catapult: introduce HGTESTCATAPULTSERVERPIPE to control run-tests' tracing If unset, it will inherit its value from HGCATAPULTSERVERPIPE (which hg itself also respects). By setting only HGTESTCATAPULTSERVERPIPE, we can get per-command breakdowns of the test runtimes for the whole test suite without overloading the trace file with the contents of the tracing from hg (such as demandimport, etc.) Differential Revision: https://phab.mercurial-scm.org/D5216

catapult: rename 'active' to 'activetrace'; this isn't storing a boolean state (This is an attempt to make it more obvious that the indentation is correct, to prevent a repeat of 97f52862b1bd) Differential Revision: https://phab.mercurial-scm.org/D5215

catapult: fix broken run-tests catapult tracing Commit 97f52862b1bd thought this block was misindented, but it was not; the indentation after that commit means we never log a catapult trace for individual commands. Also, we've never emitted the "end" trace line when the test was finished, so the trace viewer claims that these "did not finish", fixing that as well. Differential Revision: https://phab.mercurial-scm.org/D5214

test: fix self._testdir to use the right mercurial library during testing Currently if you run tests from some other directory other than the `../tests/`, you will get a warning stating ``` warning: Testing with unexpected mercurial lib: mercurial (expected /tmp/hgtests.xxxxxx/install/lib/python/mercurial) ``` This is because the current directory being added to the 'PATH', if the `self._testdir != runtestdir`, owing to this line ``` if self._testdir != runtestdir: path = [self._testdir] + path ``` Also say you ran the tests from the hg base directory, because directory is being added in the PATH (see the above snippet, at that stage the `self._testdir` has the value as `cwd`, owing to a faulty initialization). And since the current directory already has the 'hg', that is used in place of the hg that is installed for the testing purposes in `/tmp/hgtests.xxxxxx/...`. Differential Revision: https://phab.mercurial-scm.org/D5199

tests: conditonalize a difference in test-merge-tools.t on Windows

statprof: clean up unicode/bytes a little I'm not really sure how this worked before, but something perturbed it and what I've got in this change I believe is a little tidier. This fixes test-profile.t on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5210

statprof: add a couple of asserts to avoid storing unicodes I don't feel strongly about this change, so it'd be fine to drop it during review, but it was _extremely_ helpful in drafting the next change. Differential Revision: https://phab.mercurial-scm.org/D5209

tests: add some helpful `|| cat` bits to test-profile.t This way if the hg invocation crashes, you get to see a stacktrace without having to edit the test. Differential Revision: https://phab.mercurial-scm.org/D5208

tests: skip wireproto clientreactor tests on Python 3.6.0-3.6.3 inclusive See comment next to the if statement for sad details. Differential Revision: https://phab.mercurial-scm.org/D5207

contrib: tweak import-checker to always use bytes for module names We could go the other route and make embedded() return this as a bytes, but then it would be harder to portably doctest, so I prefer this option. Differential Revision: https://phab.mercurial-scm.org/D5206

contrib: fix import-checker to not b'' module names on Python 3 Caught by the doctests. Differential Revision: https://phab.mercurial-scm.org/D5205

tests: add a critical flush() to run-tests.py to make output stable on py3 Differential Revision: https://phab.mercurial-scm.org/D5204

merge-tools: when calling external merge tool, describe the resolve inputs It is a common complaint that a user will be running some operation (histedit, rebase, evolve, etc.), get into a merge-conflict situation, and not understand what they are seeing - it is possible that the merge tool is configured to display the hash, but it's difficult for most merge tools to display a good snippet of the description. In the worst case, configuring this template will lead to output that is immediately covered by a terminal application, maybe the user can hit ctrl-z to see it. In the common case, the output will be in a terminal window and a GUI program will start, and it should be possible to view both the terminal and the GUI program at the same time. Differential Revision: https://phab.mercurial-scm.org/D5094

templatekw: deprecate p1rev/p2rev/p1node/p2node in favor of p1/p2

templatekw: add p1/p2 keywords which switches the current ctx This is just an example of mappingdict. I have no idea what should be displayed as {p1|json}. Currently it is an empty dict since {'ctx'} isn't displayable, which is clearly useless and should be changed later.

templater: add wrapper for a single template mapping This can be used to nest template mappings without inserting a sequence-like layer. See the next patch for example.

templatekw: extract internal "{rev}:{node|formatnode}" template to constant This will be used later.

tweakdefaults: remove commands.resolve.mark-check=abort, it is too broken See issue6020 for the current case. I don't want to continue attempting to fix this on the stable branch, so I'm removing from tweakdefaults and will send fixes meant for the default branch and 4.9. Differential Revision: https://phab.mercurial-scm.org/D5225

resolve: when resolve.mark-check=abort, downgrade to warning if pats specified Previously, with --config resolve.mark-check=abort, running `hg resolve -m foo` would abort and emit a message saying to use --all. This command does not work, though: `hg resolve -m foo --all`, and it's really weird for --all to be the "--force" flag. My original goal with the option was to make it so that `hg resolve -m` (no filename arguments) was safer, which is why --all is used; in my mind, `hg resolve -m foo` should always mark it as resolved, and `--all` is how you specify "all the files", so that's why I chose `hg resolve -m --all` as the way out of `hg resolve -m` aborting. This commit makes all of this work the way it was meant to in my head :) Differential Revision: https://phab.mercurial-scm.org/D5218

test: remove changes preparing the drop of `HGMERGE` These changes are no longer necessary so we better drop them.