dispatch: pass around ui.fmsg channel This will be set by the command server. See the next patch.

test-subrepo: document why share magically works even if subrepos are missing I was confused how it's working while reviewing fb490d798be0, "share: reload repo after adjusting it in postshare()."

packaging: add Fedora 29 target Differential Revision: https://phab.mercurial-scm.org/D5254

store: pass in decoded filename to narrow matcher

test-narrow-clone-stream: include no usefncache case which is broken Since 9aeb9e2d28a7, encoded filenames are filtered by the narrow matcher, which is clearly wrong.

test-narrow-clone-stream: include uppercase letter in filename This makes encoded filenames differ from the original names.

phase: use the `phases.cmdphasenames` constant to walk available command flags This reverts 1ea6772fb415.

phases: add a list of names usable by the phase command This is useful in TortoiseHg for example, which has until recently just added all names in the `phasenames` list to the context menu that changes the phase. The ones not supported by the command would just error out.

transaction: display data about why the transaction failed to rollback We saw more of these a while back. Having more data available would be nice.

merge with stable

remotefilelog: avoid accessing repo instance after dispatch Upstream commit c5e6c1ba1c79 (hg: don't reuse repo instance after unshare(), 2018-09-12) poisoned the repo instance after unshare(). That made `hg unshare` fail with remotefilelog because we tried to close the fileserverclient after dispatch by accessing it via the repo. This patch fixes that by storing the reference to the fileserverclient at the beginning of dispatch. An analogous patch was sent for remotefilelog version in FB's hg-experimental as D5246. Differential Revision: https://phab.mercurial-scm.org/D5253

share: avoid a now-unnecessary reload of the repo hg.share() now returns an already-reloaded repo, so let's just use that. (This would have failed test-subrepo-recursion.t without the previous patch.) Differential Revision: https://phab.mercurial-scm.org/D5252

share: reload repo after adjusting it in postshare() When sharing a repo that's using remotefilelog, the update that happens at the end of the `hg share` call does not see the remote repo path that's copied in hg.postshare(). This patch reloads the repo after hg.postshare() to address that. This changes a subrepo test case. Note that `hg share -U; hg co tip` worked there before, so I don't see see why `hg share` should fail. I also don't know what a "locally referenced subrepo". So maybe this is fixing a bug? Hopefully it's not breaking something someone actually cares about at least. Maybe someone who knows and cares about subrepos can review this. Differential Revision: https://phab.mercurial-scm.org/D5251

tests: don't load strip extension via mq extension for remotefilelog tests The proper way to get the `hg strip` command has been via the "strip" extension since 2013. Differential Revision: https://phab.mercurial-scm.org/D5250

remotefilelog: remove unused configs for {data,history}pack version Differential Revision: https://phab.mercurial-scm.org/D5249

tests: remove obsolete PYTHONPATH override from remotefilelog tests They were added in https://bitbucket.org/facebook/hg-experimental/commits/fdcad37a6a68cfbaf5920f3eeaa0cc9cae42bd26, which says this: When remotefilelog moved from its own repo, the tests needed to be updated to adjust the PYTHONPATH to ensure the in-repo remotefilelog was loaded instead of the system one. This meant any local runs of remotefilelog tests would've been using the system remotefilelog unless the user had manually set the PYTHONPATH themselves. That doesn't seem relevant with remotefilelog in core. Differential Revision: https://phab.mercurial-scm.org/D5248

tests: drop obsolete "hginit" alias in remotefilelog tests Differential Revision: https://phab.mercurial-scm.org/D5247

fix: add extra field to fixed revisions to avoid creating obsolescence cycles The extra field prevents sequential invocations of fix from producing the same hash twice. Previously, this could cause problems because it would create an obsolescence cycle instead of the expected new successor. This change also adds an explicit check for whether a new revision should be committed. Until now, the code relied on memctx.commit() to quietly do nothing if the node already exists. Because of the new extra field, this no longer covers the case where we don't want to replace an unchanged node. Differential Revision: https://phab.mercurial-scm.org/D5245

cleanup: use revision numbers instead of hashes in test output This allows changes to the hashes produced by fix to not needlessly modify this area of the test. Differential Revision: https://phab.mercurial-scm.org/D5244

tests: make the commands.resolve.confirm tests not pollute the test-wide hgrc I want to add tests after these tests that create a new repo and use it, and was confused for a bit as to why they were seeing different behavior than I expected. Differential Revision: https://phab.mercurial-scm.org/D5242

tests: fix a couple typos in test-resolve.t comments and add a comment Differential Revision: https://phab.mercurial-scm.org/D5241

fix: add suboption for configuring execution order of tools This could be accomplished by using wrapper scripts, but that would diminish the usefulness of the incremental formatting logic. Configuring execution order along with other things in the hgrc is probably more convenient anyway. This change highlights some awkwardness with suboptions and default values, which should be addressed separately. Differential Revision: https://phab.mercurial-scm.org/D5237

tests: fix up some import statements caught by Python 3 I'm curious how the import checker manages to be so much more pedantic in Python 3, but not enough to bother exploring. Differential Revision: https://phab.mercurial-scm.org/D5240

tests: replace `tac` reimplementation by `sort -r` `sort -r` is better code "formatter" than `tac` since it's stable. It's also portable so we don't need to reimplement it in Python. Differential Revision: https://phab.mercurial-scm.org/D5239

tests: work around `tac` not being portable Introduce a tac.py helper and use it. Sigh. Differential Revision: https://phab.mercurial-scm.org/D5238

revlog: give formatting to clang-format This has two lines that are poorly formatted, both right after PyObject_HEAD macros. It's possible that in the future there will be a feature to tell clang-format that a token includes an end-of-line character (in this case the ;), but for now I'm willing to live with two poorly-formatted lines in a struct definition in exchange for not thinking about how this file is formatted. Differential Revision: https://phab.mercurial-scm.org/D5234

revlog: add blank line in comment to help clang-format Differential Revision: https://phab.mercurial-scm.org/D5233

revlog: add a comment to help clang-format produce less-awful results Differential Revision: https://phab.mercurial-scm.org/D5232

py3: ratchet caught two more passing tests Differential Revision: https://phab.mercurial-scm.org/D5236

perf: add a lazydeltabase option to perfrevlogwrite The option controls if we'll test the provided delta first. This is the current default behavior for unbundle.

perf: add `storage` as possible source for perfrevlogwrite This source will use the stored delta.

perf: add `parent-smallest` as possible source for perfrevlogwrite This source will use the smallest of the possible diff against parent.

perf: add `parent-2` as possible source for perfrevlogwrite This source will use a diff against p2 if it exists and fall back to p1 otherwise.

perf: add `parent-1` as possible source for perfrevlogwrite This source will use a diff against p1 in all case.

perf: add the notion of "source" to perfrevlogwrite We want to test performance associated witch various way to add a new revision. They will be specified using this new argument.

perf: only display the total time for perfrevlogwrite if quiet This provide a simple way to get an overview of the total performance.

perf: offer full details in perfrevlogwrite This will be useful for people who want to study the timing pattern more closely.

perf: introduce a perfrevlogwrite command The command record times taken by adding many revisions to a revlog. Timing each addition, individually. The "added revision" are recreations of the original ones. To time each addition individually, we have to handle the timing and the reporting ourselves. This command is introduced to track the impact of sparse-revlog format on delta computations at initial storage time. It starts with the full text, a situation similar to the "commit". Additions from an existing delta are better timed with bundles. The complaints from `check-perf-code.py` are not relevant. We are accessing and "revlog" opener, not a repository opener.

tests: fix config knob in test-narrow-clone-stream.t Two patches landed in parallel and had a semantic conflict. This resolves the mess and leaves us with passing tests. Differential Revision: https://phab.mercurial-scm.org/D5231

remotefilelog: fix various whitespace issues in docstring Differential Revision: https://phab.mercurial-scm.org/D5230

ui: add config knob to redirect status messages to stderr (API) This option can be used to isolate structured output from status messages. For now, "stdio" (stdout/err pair) and "stderr" are supported. In future patches, I'll add the "channel" option which will send status messages to a separate command-server channel with some metadata attached, maybe in CBOR encoding. This is a part of the generic templating plan: https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Sanity_check_output .. api:: Status messages may be sent to a dedicated stream depending on configuration. Don't use ``ui.status()``, etc. as a shorthand for conditional writes. Use ``ui.write()`` for data output.

ui: hide fin/fout/ferr attributes behind @property functions This allows keeping references to fout/ferr/fin which are updated when these properties are changed. See the next patch.

ui: label prompt and echo messages I'm going to add a dedicated command-server channel for status messages, which carries metadata alongside a message text. 'ui.*' label provides a hint how message text should be processed.

ui: add inner function to select write destination I'm going to add a config knob to redirect any status messages to stderr. This function helps to switch underlying file objects. # no-check-commit because of existing write_err() function

ui: remove _write() and _write_err() functions

ui: move pre/post processes from low-level write()s to _writenobuf() This helps adding a dedicated stream for status/error messages. I don't want to add _write*() function per stream.

ui: pass in file object to _writenobuf() See the subsequent patches for why. The "if" block in _writenobuf() will be removed soon.

perf: fix perfrevlogrevisions --reverse Currently, 'endrev' equals `len(revlog)`, a revision that does not exist. When asking for the reverse order, the arguments passed to xrange are `xrange(len(revlog), startrev)` which then crash. We need to offset 'endrev' by one so we don't crash anymore. Also, we offset 'startrev' to ensure we get the same number of revisions with and without the `--reverse` option. Differential Revision: https://phab.mercurial-scm.org/D5228

procutil: import concerns about creationflags on Windows from D1701 I don't have the need anymore for the change in D1701 nor the time to investigate the changes on all supported Windows platforms. I import the stuff I learned on D1701 in the `runbgcommand` so the next people working on it can starts from there. Differential Revision: https://phab.mercurial-scm.org/D5229

localrepo: extract loading of hgrc files to standalone function Various 3rd party extensions supplement where per-repo config data lives. Looking at their sources, they resort to unorthodox means to inject the config data. And the way they do it is susceptible to corner cases. e.g. not processing automatic extension loads, not reacting to new or disabled extensions in configs, etc. This commit extracts the core logic of loading hgrc files into a standalone function so there is a clear function that can be monkeypatched to inject per-repo config data at repository open time. Differential Revision: https://phab.mercurial-scm.org/D5221

revsets: make bookmark/named('re:nonexistent') not abort (issue6018) (BC) Foozy documented the differences between revsets branch(), tag(), bookmark(), and named() in eeb5d5ab14a6 (revset: raise RepoLookupError to make present() predicate continue the query, 2015-01-31). He seemed to want tag() to change behavior to not error out on non-matching regular expressions. I think it's instead bookmark() and named() that should not error out. So that's what this patch does. Differential Revision: https://phab.mercurial-scm.org/D5220

fix: rename :fileset subconfig to :pattern This name was always inaccurate, since the config accepts any pattern. Hopefully so few people use this right now that it won't matter, but there will now be a warning if the old config name is used. Differential Revision: https://phab.mercurial-scm.org/D5226

fix: add a config to abort when a fixer tool fails This allows users to stop and address tool failures before proceeding, instead of the default behavior of continuing to apply any tools that didn't fail. For example, a code formatting tool could fail if you have syntax errors, and you might want your repo to stay in its current state while you fix the syntax error before re-running 'hg fix'. It's conceivable that this would even be necessary for the correctness of some fixer tools across a chain of revisions. Differential Revision: https://phab.mercurial-scm.org/D5200

perf: measure slicing time in perfrevlogrevision Slicing a sparse delta chain can be expensive. We now benchmark the associated time.

perf: teach perfrevlogrevision about sparse reading Before this change, chunks were always read in a single block. Even in the sparse-read/sparse-revlog case. This gave a false view of the performance and could lead to memory consumption issue.

perf: use the same timer for all section of perfrevlogrevision Otherwise the -T json output is invalid.

obsutil: clarify the access to "repo" We use the variable multiple times and we might use it even more in the future. We use a temporary variable instead.

pycompat: adding Linux detection and fixing Mac Python 3 recommends detecting OSs with the prefix of the platform, but we were comparing the full string for macOS. We also didn't have Linux detection, which is convenient for extensions to use (rather than have some OSs detected by hg and some by the extension). Reference: https://docs.python.org/3/library/sys.html#sys.platform Differential Revision: https://phab.mercurial-scm.org/D5227

remotefilelog: fix various minor py3 problems # skip-blame b prefixes and pycompat.long, nothing remotely interesting Differential Revision: https://phab.mercurial-scm.org/D5223

remotefilelog: rip out a missed mention of lz4 Differential Revision: https://phab.mercurial-scm.org/D5222

ui: wrap whole _write() block with timeblockedsection I think the cost of color labeling is negligible compared to the I/O syscalls. Let's simply wrap the whole write() function so that we can eliminate _write() and _write_err() in later changeset.

ui: indent _writenobuf() to prepare moving bits from _write() functions

ui: simply concatenate messages before applying color labels This should be cheaper in space than applying labels for each message.

ui: simplify interface of low-level write() functions _write() and _write_err() will be replaced with fout.write() and ferr.write() respectively. This is the first step.

ui: factor out function that writes data to fout/ferr with labeling I'm thinking of adding an option to send status messages to stderr (or a dedicated command-server channel) so that structured output (e.g. JSON) would never be interleaved with non-formatter output. A unified write() interface helps to do that.

ui: consolidate places where _progclear() is called The progress bar has to be cleared when we start writing some data to the output stream. Let's make it always triggered immediately before switching by _colormode, so that we can easily factor out helper functions.

templater: compute revset lazily This speeds up e.g. "{ifcontains(rev, revset('::.'), ...)}" in common cases where 'rev' is near the working parent. The templater API is ugly, but it helps here. 'f' can be either a generator or a function returning a generator.

remotefilelog: drop compat code for "getbundle_shallow" wireprotocol command Doing some annotate on hgexperimental shows that getbundle_shallow used to exist in 2013 or before. We don't have any pre-2013 remotefilelog users except Fb themselves and I doubt they are going to use in-core remotefilelog. So it's safe to remove this. Differential Revision: https://phab.mercurial-scm.org/D5193

tests: remove lz4 as dependency while running tests One of the previous patch authored by Augie rips out the lz4 dependency and things should work without it. Now there are just 2-3 tests failing because of same change in emitrevisions() API. Differential Revision: https://phab.mercurial-scm.org/D5192

remotefilelogserver: add a matcher argument to _walkstreamfiles() Implementing narrow stream clones in core, I added an optional matcher argument. The function in remotefilelogserver.py does not know about that argument and does not accept that and hence some tests fails. Differential Revision: https://phab.mercurial-scm.org/D5191

shallowutil: introduce a helper function isenabled() This patch introduces a inenabled() function which will check whether remotefilelog is enabled or not. The function is then also used at all the places where check whether remotefilelog is enabled or not. The new function makes code easy to read without need to understand what is the constant involved and why we are checking repo.requirements. Differential Revision: https://phab.mercurial-scm.org/D5190

remotefilelog: remove some BC code related to streamclone Since remotefilelog is now moved to core and we have streamclone._walkstreamfiles() in core, we don't need to have the backward compatibility code. People with old mercurial version should use remotefilelog from hg-experimental repo as IMO remotefilelog will go under a good refactoring and old clients will break. # no-check-commit foo_bar function name Differential Revision: https://phab.mercurial-scm.org/D5189

remotefilelog: add some docstring Differential Revision: https://phab.mercurial-scm.org/D5131

remotefilelog: rename wireproto methods and remaining capabilities This is a mess, in part because there should be more constants throughout. I know we typically do exp- instead of the x_ business in this change, but I also had to use this in some function names, so I figured until I can break that coupling I'd go with this. If it's too unpleasant during review, let me know and I can probably clean it up some more. # no-check-commit due to new foo_bar naming - too hard to avoid right now :( Differential Revision: https://phab.mercurial-scm.org/D5129

remotefilelog: consolidate and rename on-disk store requirement The value of this constant appeared in too many places. While we're here, rename it to be more consistent with our naming conventions for experimental functionality. Differential Revision: https://phab.mercurial-scm.org/D5128

remotefilelog: consolidate and rename bundle2 capability Bonus: we catch a spot where the shallowrepo requirement was being misused as the bundle2 capability. Differential Revision: https://phab.mercurial-scm.org/D5127

remotefilelog: rename capability for legacy ssh file fetching method I think I want to delete this, but for now we'll just rename this so it's easy to tell apart from other strings that have the same value. Differential Revision: https://phab.mercurial-scm.org/D5126

remotefilelog: rip out lz4 support Some methods are left teased out so it's easier to build a migration extension from v1 packs to v2. I also anticipate those methods will be of use if we get around to adding configurable compression engines before we end up jettisoning packs. I think it would make sense to register one-shot `compress` and `decompress` methods on our compression engines in util.py, but indygreg mentioned that might not be straightforward. Differential Revision: https://phab.mercurial-scm.org/D4944

remotefilelog: bump pack file formats to version 2 Drop support for format 0 and 1 packs. Version 2 will be different in that it'll use zlib compression instead of lz4, as the latter isn't really suitable for core. If I can make it work, I'll make compression engines pluggable, but the logic for handling these files is spread out enough it's a little tricky to get that right. Test changes are only due to pack files being renamed. The contents of the packs are the same. Differential Revision: https://phab.mercurial-scm.org/D4943

remotefilelog: avoid hard-coding pack names in tests I'm about to alter the pack format, which alters the content-addressed name of the pack file. In order to reduce my own confusion I found it helpful to use an ls | head pipeline since we always want to inspect the newest pack. Differential Revision: https://phab.mercurial-scm.org/D4942

remotefilelog: remove now-unused wirepack code Differential Revision: https://phab.mercurial-scm.org/D5124

remotefilelogserver: remove pack-serving functionality Differential Revision: https://phab.mercurial-scm.org/D5123

remotefilelog: remove support for fetching packs We've never used this at Google, and it's strictly an optimization that indygreg and I are confident we can handle more elegantly later. Let's rip it out to keep the surface area of the imported remotefilelog small. Some tests have changes because the tests for regular packs and repacking were mixed with tests of fetching packs. I'm pretty sure I've done the right thing here. Differential Revision: https://phab.mercurial-scm.org/D5122

remotefilelog: jettison example cache client We can always reconstruct it from history if it becomes interesting. Differential Revision: https://phab.mercurial-scm.org/D4941

remotefilelog: prune obsolete method Differential Revision: https://phab.mercurial-scm.org/D4940

logtoprocess: use new runbgcommand from procutil Differential Revision: https://phab.mercurial-scm.org/D4939

procutil: port over windows encoding fixes from logtoprocess Differential Revision: https://phab.mercurial-scm.org/D5121

remotefilelog: transplant runbgcommand to procutil While cleaning up the deprecated runshellcommand I noticed a near-clone of this in logtoprocess, so I'm standardizing on what appears to be the newer one by moving it to procutil. Differential Revision: https://phab.mercurial-scm.org/D4938

remotefilelog: remove function that was described as deprecated Differential Revision: https://phab.mercurial-scm.org/D4937

remotefilelog: import pruned-down remotefilelog extension from hg-experimental This is remotefilelog as of my recent patches for compatibility with current tip of hg, minus support for old versions of Mercurial and some FB-specific features like their treemanifest extension and fetching linkrev data from a patched phabricator. The file extutil.py moved from hgext3rd to remotefilelog. This is not yet ready to be landed, consider it a preview for now. Planned changes include: * replace lz4 with zstd * rename some capabilities, requirements and wireproto commands to mark them as experimental * consolidate bits of shallowutil with related functions (eg readfile) I'm certainly open to other (small) changes, but my rough mission is to land this largely as-is so we can use it as a model of the functionality we need going forward for lazy-fetching of file contents from a server. # no-check-commit because of a few foo_bar functions Differential Revision: https://phab.mercurial-scm.org/D4782

store: introduce _matchtrackedpath() and use it to filter store files This patch introduces a function to filter store files on the basis of the path which they are tracking. The function assumes that the entries can be of two types, 'meta/*' and 'data/*' which means it will just work on revlog based storage and not with another storage ways. For the 'data/*' entries, we remove the 'data/' part and '.i/.d' part from the beginning and the end then pass that to matcher. For the 'meta/*' entries, we remove the 'meta/' and '/00manifest.(i/d)' part from beginning and end then call matcher.visitdir() with it to make sure all the parent directories are also downloaded. Since the storage filtering for narrow stream clones is implemented with this patch, we remove the un-implemented error message, add some more tests and add the treemanifest case to tests too. The tests demonstrate that it works correctly. After this patch, we have now narrow stream clones working. Narrow stream clones are a very important feature for large repositories who have good internet connection because they use streamclones for cloning and if they do normal narrow clone, that takes more time then a full streamclone. Also narrow-stream clone will drastically speed up clone timings. Differential Revision: https://phab.mercurial-scm.org/D5139

narrow: only send includepats and excludepats if they are not empty If we send an empty includepats or excludepats argument to getbundle, it's translated to `['']` on the server which causes problems because even though it's empty, bool of that value if True and we end up creating differencematcher with narrowspec.match() which results in unexpected behavior. Differential Revision: https://phab.mercurial-scm.org/D5138

exchange: pass includepats and excludepats as arguments to getbundle() This will help in implementing narrow stream clones. Also narrow extension used to add these arguments, now we add them by default if they are not empty. Since reading includepats and excludepats on the server only works when narrow is enabled, we check if narrow if enabled or not before passing them. Differential Revision: https://phab.mercurial-scm.org/D5119

catapult: add a bit more documentation on how to use catapult tracing Differential Revision: https://phab.mercurial-scm.org/D5217

catapult: introduce HGTESTCATAPULTSERVERPIPE to control run-tests' tracing If unset, it will inherit its value from HGCATAPULTSERVERPIPE (which hg itself also respects). By setting only HGTESTCATAPULTSERVERPIPE, we can get per-command breakdowns of the test runtimes for the whole test suite without overloading the trace file with the contents of the tracing from hg (such as demandimport, etc.) Differential Revision: https://phab.mercurial-scm.org/D5216

catapult: rename 'active' to 'activetrace'; this isn't storing a boolean state (This is an attempt to make it more obvious that the indentation is correct, to prevent a repeat of 97f52862b1bd) Differential Revision: https://phab.mercurial-scm.org/D5215

catapult: fix broken run-tests catapult tracing Commit 97f52862b1bd thought this block was misindented, but it was not; the indentation after that commit means we never log a catapult trace for individual commands. Also, we've never emitted the "end" trace line when the test was finished, so the trace viewer claims that these "did not finish", fixing that as well. Differential Revision: https://phab.mercurial-scm.org/D5214

test: fix self._testdir to use the right mercurial library during testing Currently if you run tests from some other directory other than the `../tests/`, you will get a warning stating ``` warning: Testing with unexpected mercurial lib: mercurial (expected /tmp/hgtests.xxxxxx/install/lib/python/mercurial) ``` This is because the current directory being added to the 'PATH', if the `self._testdir != runtestdir`, owing to this line ``` if self._testdir != runtestdir: path = [self._testdir] + path ``` Also say you ran the tests from the hg base directory, because directory is being added in the PATH (see the above snippet, at that stage the `self._testdir` has the value as `cwd`, owing to a faulty initialization). And since the current directory already has the 'hg', that is used in place of the hg that is installed for the testing purposes in `/tmp/hgtests.xxxxxx/...`. Differential Revision: https://phab.mercurial-scm.org/D5199

tests: conditonalize a difference in test-merge-tools.t on Windows

statprof: clean up unicode/bytes a little I'm not really sure how this worked before, but something perturbed it and what I've got in this change I believe is a little tidier. This fixes test-profile.t on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5210

statprof: add a couple of asserts to avoid storing unicodes I don't feel strongly about this change, so it'd be fine to drop it during review, but it was _extremely_ helpful in drafting the next change. Differential Revision: https://phab.mercurial-scm.org/D5209

tests: add some helpful `|| cat` bits to test-profile.t This way if the hg invocation crashes, you get to see a stacktrace without having to edit the test. Differential Revision: https://phab.mercurial-scm.org/D5208

tests: skip wireproto clientreactor tests on Python 3.6.0-3.6.3 inclusive See comment next to the if statement for sad details. Differential Revision: https://phab.mercurial-scm.org/D5207

contrib: tweak import-checker to always use bytes for module names We could go the other route and make embedded() return this as a bytes, but then it would be harder to portably doctest, so I prefer this option. Differential Revision: https://phab.mercurial-scm.org/D5206

contrib: fix import-checker to not b'' module names on Python 3 Caught by the doctests. Differential Revision: https://phab.mercurial-scm.org/D5205

tests: add a critical flush() to run-tests.py to make output stable on py3 Differential Revision: https://phab.mercurial-scm.org/D5204

merge-tools: when calling external merge tool, describe the resolve inputs It is a common complaint that a user will be running some operation (histedit, rebase, evolve, etc.), get into a merge-conflict situation, and not understand what they are seeing - it is possible that the merge tool is configured to display the hash, but it's difficult for most merge tools to display a good snippet of the description. In the worst case, configuring this template will lead to output that is immediately covered by a terminal application, maybe the user can hit ctrl-z to see it. In the common case, the output will be in a terminal window and a GUI program will start, and it should be possible to view both the terminal and the GUI program at the same time. Differential Revision: https://phab.mercurial-scm.org/D5094

templatekw: deprecate p1rev/p2rev/p1node/p2node in favor of p1/p2

templatekw: add p1/p2 keywords which switches the current ctx This is just an example of mappingdict. I have no idea what should be displayed as {p1|json}. Currently it is an empty dict since {'ctx'} isn't displayable, which is clearly useless and should be changed later.

templater: add wrapper for a single template mapping This can be used to nest template mappings without inserting a sequence-like layer. See the next patch for example.

templatekw: extract internal "{rev}:{node|formatnode}" template to constant This will be used later.

test: remove changes preparing the drop of `HGMERGE` These changes are no longer necessary so we better drop them.

run-tests: define the default merge tool through configuration Using the `HGMERGE` environment variable generates confusion as it overrides any value set through configuration. By setting the default value through the default HGRC, the usual overriding rules apply and it should reduce confusion.

run-tests: explicitly declare the list of dropped environment variable It will make the list clearer and more maintainable.

test: drop usage of `HGMERGE` in `test-lfconvert.t` Using configuration instead of environment variable has a strange effect on the output, but no impact on the end result.

test: drop usage of `HGMERGE` in `test-merge-types.t` Using configuration instead of environment variable has a strange effect on the output, but no impact on the end result.

test: use the "correct" merge tool in `test-rebase-detach.t` The value of `HGMERGE` overrides the value passed through --config. We are about to drop this `HGMERGE` environment variable so we update the value used in config to match what is currently in use. This is another example of confusion introduced by the `HGMERGE` environment variable.

test: also deactivate `ui.merge` when testing merge tool We are about to introduce a default config for `ui.merge`. Since `test-merge-tools.t` is about testing merge configuration, we need to make sure it won't interfere.

tests: adjust merge tool config in test-largefiles-update.t Previously, the `ui.merge=internal:fail` configuration was shadowed by the the `HGMERGE=internal:merge` environment variable. We would like to remove this usage of `HGMERGE` variable because it is error-prone (eg: this very test). We start by cleaning up test independently to avoid large churn when the change happens.

extensions: include current version in "invalid version" message It is "not so rare" for the mercurial version to be badly detected at build time. In such case, version check for extensions gets confused. To help pinpoint the error, we now include the Mercurial's version number in the error message.

obsolete: prefetch the repo.obsstore used in phasedivergence loop The speedup is probably quite negligible, but it cannot hurt.

obsutil: prefetch method in allpredecessors loop We don't expect a massive speedup from this, but the change was laying around in my repository and it cannot hurt.

merge with stable

beautifygraph: don't substitute anything for 'X' in rendered graphs It looks like we never actually render graphs this way, although there's an example in a comment next to the code that prevents it from being needed (see graphmod.ascii()). Differential Revision: https://phab.mercurial-scm.org/D5103

py3: do not stringify integers in revlog stats by '%s'

py3: convert revlog stats to a dict of (bytes, int) pairs Py_DECREF(t) is replaced with Py_CLEAR(t) so that t is set to NULL once decrefed. Otherwise, it would be excessively decrefed if a subsequent PyBytes_FromString() failed.

py3: roll up threading.Thread constructor args into **kwargs The constructor doesn't have a `verbose` keyword argument in py3.

tests: glob over a single quote vs double quote difference on Windows

py3: port test-log-exthook.t to Python 3 For once an easy one. Differential Revision: https://phab.mercurial-scm.org/D5184

extensions: fix up many many debug logs that use %r This gets us a bunch closer on Python 3, but I'll still have to use a ton of sad globs. A previous version of this patch tried to preserve the %r formatting, but upon review Yuya noted that special characters in extension names is very unlikely, so we can just use %s and not sweat the quoting. Differential Revision: https://phab.mercurial-scm.org/D5002

blackbox: add configitem for format of log timestamps Sometimes blackbox logs are used to report performance problems, but the timestamps are only at second granularity, so often the timings have to stated separately by the reporter. This is inconvenient and error-prone, so I would like to include %f in the date format. This patch makes that possible.

narrow: replace filtering in list comprehension by set operations I didn't think of this while reviewing the patch. Differential Revision: https://phab.mercurial-scm.org/D5188

storage: update sqlitestore to use the new `deltamode` parameter While updating the filelog class, I forget to update the sqlitestore, tests are now passing with this patch. Differential Revision: https://phab.mercurial-scm.org/D5185

narrow: rework logic to check whether we need to widen and narrow This patch reworks logic which calculates whether we need to extend or narrow our working copy or not. We filter the addincludes, removeincludes, addexcludes and removeexcludes passed from user to the actual added and removed includes and excludes. What that means is a user can pass an already included path as addincludes, a path which is not included as removeincludes etc. In such situations the old logic use to think we need to do some work, whereas we don't need to do that work. In old logic, even if we don't have anything new to include but it believes we need to call widen, this adds some good amount of work on large repository. A widen calls involves computing incomming csets, calling the narrow_widen() which in non-ellipses cases goes through all the set of csets which are available which can take ~2-3 mins on large repos. Those 2-3 minutes are spend on doing nothing which a client can prevent by checking is there really anything which needs to be included. The tests changes shows that we don't go to the server anymore in such cases which is nice. Differential Revision: https://phab.mercurial-scm.org/D5183

tests: show that adding an already included path still calls narrow_widen() This patch adds tests demonstrating that we still go to the server in non-ellipses widening when we have that path already on the client and there is nothing new to download. The next patch will try to make client side logic smart and not go to the server if we don't need to download anything. Differential Revision: https://phab.mercurial-scm.org/D5182

graft: introduce --base option for using custom base revision while merging The graft command usually performs an internal merge of the current parent revision with the graft revision, using p1 of the grafted revision as base for the merge. As a trivial extension of this, we introduce the --base option to allow for using another base revision. This can be used as a building block for grafting and collapsing multiple changesets at once, or for grafting the resulting change from a merge as a single simple change. (This is kind of similar to backout --parent ... only different: this graft base must be an ancestor, but is usually *not* a parent.) This is probably an advanced use case, and we do thus not show it in the non-verbose help.

changegroup: add a option to create bundle with full snapshot only This is easy to implement now and can be useful for benchmarking.

changegroup: allow to force delta to be against p1 This new developer option is useful to general more "generic" bundle. Without this option, a bundle generated from the repository use deltas similar to the one stored in the specific repository it was generated from. This makes performance testing a bit tricky. Using deltas similar to the final result means all delta stored in the bundle can be applied to the target repository without any further processing (except for the rare case of a full snapshot). The application of such bundles (almost) never exercises the (slower) path of searching for a new valid delta. This result in unrealistic and too favorable timing and profile. Instead, we introduce an option to make sure all revisions are stored as a delta against p1. It might not be the best generation option, but it guarantees that the content will be "generic", not favoring a specific target.

storage: also use `deltamode argument` for ifiledata Now that lower level uses such argument, we can propagate the change to higher layers.

changegroup: refactor emitrevision to use a `deltamode` argument This new argument gathers the semantic of `sendfulltext` and `deltaprevious` in a single value. We are about to introduce a new type of constraints. Avoiding yet another argument sounds like a plus.

branchmap: do not specify changelog as an argument Since (unfiltered)repo.changelog lookup gets as fast as __dict__ lookup, there's no point to pass in changelog instance. $ hg perfbranchmap --clear-revbranch -R mozilla-central ! base (orig) wall 20.593091 comb 20.600000 user 20.520000 sys 0.080000 (best of 3) (this) wall 20.129126 comb 20.130000 user 20.020000 sys 0.110000 (best of 3) This backs out most of the changes in 76d4272bd57b and 47c03042cd1d.

filecache: unimplement __set__() and __delete__() (API) Implementing __set__() implies that the descriptor can't be overridden by obj.__dict__, which means any property access involves slow function call. "Data descriptors with __set__() and __get__() defined always override a redefinition in an instance dictionary. In contrast, non-data descriptors can be overridden by instances." https://docs.python.org/2.7/reference/datamodel.html#invoking-descriptors This patch basically backs out 236bb604dc39, "scmutil: update cached copy when filecached attribute is assigned (issue3263)." The problem described in issue3263 (which is #3264 in Bugzilla) should no longer happen since repo._bookmarkcurrent has been moved to repo._bookmarks.active. We still have a risk of introducing similar bugs, but I think that's the cost we have to pay. $ hg perfrevset 'branch(tip)' -R mercurial (orig) wall 0.139511 comb 0.140000 user 0.140000 sys 0.000000 (best of 66) (prev) wall 0.114195 comb 0.110000 user 0.110000 sys 0.000000 (best of 81) (this) wall 0.099038 comb 0.110000 user 0.100000 sys 0.010000 (best of 93)

filecache: use try-except for faster __dict__ lookup Python function call is slow, and the cost could be significant here. $ hg perfrevset 'branch(tip)' -R mercurial (orig) wall 0.139511 comb 0.140000 user 0.140000 sys 0.000000 (best of 66) (this) wall 0.114195 comb 0.110000 user 0.110000 sys 0.000000 (best of 81)

help: displaying extension commands by default Differential Revision: https://phab.mercurial-scm.org/D5156

help: displaying documented aliases by default This makes aliases be displayed in "hg help" when they have a :doc config entry, and also allows them to be assigned to a category with :category. Differential Revision: https://phab.mercurial-scm.org/D5087

help: allow hiding of help topics Differential Revision: https://phab.mercurial-scm.org/D5077

help: allow commands to be hidden This is useful in enterprise environments where some workflows are discouraged. Differential Revision: https://phab.mercurial-scm.org/D5076

py3: add one more passing test to whitelist Caught by python 3 builder. Differential Revision: https://phab.mercurial-scm.org/D5175

py3: make sure we pass sysstr in sqlite3.connect() Differential Revision: https://phab.mercurial-scm.org/D5174

archive: use manifest.matches() to simplify and speed up matching manifest.matches() can avoid walking paths the user did not want to archive. Differential Revision: https://phab.mercurial-scm.org/D5178

archive: create alwaysmatcher when no matcher provided Differential Revision: https://phab.mercurial-scm.org/D5177

archive: change "matcnfn" argument to a real matcher All callers seem to be passing a real matcher, not just a function. We were also passing it into match.subdirmatcher(), which assumes it is a matcher. Differential Revision: https://phab.mercurial-scm.org/D5176

histedit: crashing with a more useful error message on empty defaultrev Before this, `hg --config histedit.defaultrev= histedit` would crash with File "destutil.py", line 385, in desthistedit if revs: UnboundLocalError: local variable 'revs' referenced before assignment Differential Revision: https://phab.mercurial-scm.org/D5543

Added signature for changeset 197f092b2cd9

Added tag 4.8.2 for changeset 197f092b2cd9

server: always close http socket if responding with an error (issue6033) It's possible for hgweb to respond _very_ early with an error if we're catching certain types of errors. When we do, we need to tell the client the socket is toast when there's a POST involved because otherwise there can be lingering POST data on the socket that will confuse any future requests on the socket. This manifested as a flaky failure on Linux in an lfs extension test and a reliable failure on FreeBSD. With this patch applied, test-lfs-serve-access.t now passes for me on FreeBSD. Differential Revision: https://phab.mercurial-scm.org/D5498

match: fix assertion for fileset with no context (issue6046) A falsy changectx should be allowed.

templatekw: fix documentation typos

update: do not pass in user revspec as default destination (issue6044) When the revsingle() was introduced at 61c0df2b089a, it couldn't handle revspec=0 (not '0') properly. That's probably why the default was set to rev. This is technically BC since "hg update ''" was identical to "hg update '.'" whereas "hg update -r ''" is "hg update", but I believe that's a bug given no test fails with this change.

revlog: cache delta base value under -1 Such base are invalid so we better report them early.

revlog: catch revlog corruption in index_baserev A revision cannot use a base above itself, it can only happens one corrupted repository. Ignoring such corrupted could lead to infinite loop.

phabricator: properly encode boolean types in the request body I tripped over this playing with `hg debugcallconduit` to query for valid reviewers. If the JSON on stdin is written as 'True' or 'False', python complains it isn't valid JSON. If it's written as 'true' or 'false', it made it to the server, but got kicked back with this: abort: Conduit Error (ERR-CONDUIT-CORE): Error while reading "isBot": Expected boolean (true or false), got something else. The test isn't really relevant here (the code can be reverted, and it will pass), but this gives us coverage for the debug command.

worker: do not swallow exception occurred in main process Before, SystemExit(255) would be most likely raised since the worker processes were terminated by the main process and the status would be set to 255 in response. We should instead re-raise the exception occurred first. It's pretty hard to debug problems like the issue 6035 with no traceback.

run-tests: fix permission to clean up unreadable directories I found many hgtests.* directories left in $TMPDIR, which couldn't be deleted because test-ssh-repoerror.t created some directories with a-rx mode.

context: reimplement memfilectx.cmp() If I added a sanity check to basefilectx, test-context.py exploded. This patch copies the naive implementation from overlayworkingfilectx.

windows: ensure pure posixfile fd doesn't escape by entering context manager There are tests in test-revlog-mmapindex.t and test-rebase-mq-skip.t that are fixed by this, but we usually don't use --pure on Windows. For whatever reason, the remaining --pure failures are various errors like $ENOTDIR$ and "Access is denied" have a trailing '.'.

vfs: ensure closewrapbase fh doesn't escape by entering context manager I'm not sure if there's a problem in practice here, as there's no test failure either way. The __exit__() and close() methods raise an exception, so maybe __exit__() and close() are being called directly on the underlying handle when delayclosedfile is used on a context manager? I doubt that was intended.

windows: ensure mixedfilemodewrapper fd doesn't escape by entering context mgr Otherwise it seems that the special read and write handling would be bypassed.

py3: ensure the proxied Windows fd doesn't escape by entering context manager The purpose of the proxy class is to provide the `name` attribute which contains the file path. But in tests that used a context manager, it still blew up complaining that 'int' doesn't have a 'startswith' function.

test: fix test-http-bad-server with current python 2.7 https://github.com/python/cpython/pull/2825 changed the exception message for empty http status line. Differential Revision: https://phab.mercurial-scm.org/D5412

hgweb: register web.comparisoncontext to the config table This was caught in some server side logging added to debug py3 issues.

Added signature for changeset 1c8c54cf9725

Added tag 4.8.1 for changeset 1c8c54cf9725

rebase: fix path auditing to audit path relative to repo root (issue5818) Before this patch, when rebasing a file called "foo/bar", we would check e.g. if "/foo" (i.e. rooted at the file system root) was a symlink. Differential Revision: https://phab.mercurial-scm.org/D5361

tests: show bad path auditing in in-memory rebase Thanks to Yuya for providing this test case in https://bz.mercurial-scm.org/show_bug.cgi?id=5818. Differential Revision: https://phab.mercurial-scm.org/D5368

tests: add a missing "cd .." to test-rebase-inmemory.t Differential Revision: https://phab.mercurial-scm.org/D5367

rust: fix possible out-of-bounds read through index_get_parents() index_get_parents() is an internal function, which doesn't check if the specified rev is valid. If rustlazyancestors() were instantiated with an invalid stoprev, it would access to invalid memory region. This is NOT a security fix as there's no Python code triggering the bug, but included in this series to not give a notion about the memory issue fixed by the previous patch.

revlog: fix out-of-bounds access by negative parents read from revlog (SEC) 82d6a35cf432 wasn't enough. Several callers don't check negative revisions but for -1 (nullrev), which would directly lead to out-of-bounds read, and buffer overflow could follow. RCE might be doable with carefully crafted revlog structure, though I don't think this would be useful attack surface.

rebase: fix dir/file conflict detection when using in-mem merge Differential Revision: https://phab.mercurial-scm.org/D5360

tests: show that in-mem rebase does not find path dir/file conflicts Differential Revision: https://phab.mercurial-scm.org/D5359

extdiff: register the configuration generated commands with a help category Otherwise, 'extdiff' shows up under file management and the rest of the commands are at the bottom under 'Uncategorized'.

rebase: abort in-mem rebase if there's a dirty merge state In-memory merge uses the on-disk merge state, so we should not allow it run in-memory merge when the merge state is not clean. We should probably not use the on-disk merge state when running in-memory merge, but chaning that is not suitable for the stable branch. Differential Revision: https://phab.mercurial-scm.org/D5357

rebase: preserve working copy when redoing in-mem rebase on disk When in-memory rebase runs into conflicts, we retry it on disk. But before we do that, we abort the in-memory rebase. That is done because even though it's mostly in memory, there are still a few state files written (e.g. the merge state). We should make it not write those files so we don't need to abort, but for the stable branch, let's explicitly clear the state we need to clear instead of running the usual abort code. Differential Revision: https://phab.mercurial-scm.org/D5356

tests: show that in-mem rebase falling back loses state Both working copy changes and the merge state is lost when in-memory rebase falls back to on-disk mode. Differential Revision: https://phab.mercurial-scm.org/D5355

commandserver: get around ETIMEDOUT raised by selectors2 selector.select() should exits with an empty event list on timed out, but selectors2 raises OSError if timeout expires while recovering from EINTR. Spotted while debugging new chg feature.

selectors2: backport minimal fix of timeout handling from 2.0.1 The original code would raise TypeError since OSError() doesn't support keyword arguments. We can't simply import the selectors 2.0.1, which still spawns "uname -p" through platform.system(). We could switch to the unreleased version, but I decided to not right now to minimize the change.

tests: sniff for libfuzzer actually being available in test-fuzz-targets.t When I upgraded the FreeBSD buildbot to 11.2 it seems we picked up clang6, but the default clang on FreeBSD doesn't include libfuzzer. I can't find a way to sniff for libfuzzer without running a compile, so here we are. Differential Revision: https://phab.mercurial-scm.org/D5270

tests: sniff for /usr/local/bin/gmake and use it in test-fuzz-targets.t This isn't as robust as it probably should be, but for now it'll get the job done on the buildbots. Differential Revision: https://phab.mercurial-scm.org/D5269

tests: stabilize test-inherit-mode.t on FreeBSD and macOS (issue6026) Symbolic links are funny permissions-wise, but on the linked issue Yuya has convinced me that we can ignore this permissions issue on macOS (FreeBSD allows setting permissions bits but ignores them) and we'll be in fine shape.

wireprotov2peer: wait for initial object before resolving future As part of rolling out wireprotov2 with redirect support, I encountered an edge case with regards to future resolution. Essentially, the initial response frame from the server did not fully decode the initial CBOR object. The frame wasn't marked as EOS. In the previous code, we resolved the future for the request to response.objects(), which mapped to the commandresponse instance which would eventually produce a redirect. Upon receiving subsequent data, the initial CBOR object containing the redirect would be decoded and we'd process the redirect. However, the future would already have been resolved with the initial commandresponse.objects() and the client iterating over the objects wouldn't receive any objects from the redirect because the redirect was populating a different commandresponse instance! This commit changes the logic so we don't resolve futures until the initial CBOR response object is fully decoded or until EOS occurs. In cases where there is an empty or partial frame associated with a redirect, the future will now resolve with the commandresponse containing the proper series of decoded objects.

wireprotov2peer: always return a bool from _processredirect() Without this, we may stop servicing the redirect response if the future has already been resolved. And the future will often be resolved very early, since many consumers iterate the decoded CBOR object stream and expect data to lazily arrive.

tests: stabilize the recent checkexec changes on Windows This goes with bd0874977a5e.

checkexec: create destination directory if necessary Since 460733327640, a "share" use the cache of the source repository. A side effect is that no `.hg/cache` directory exists in the "share" anymore. As a result, the checkexec logic can't use it to create its temporary file and have to use the working copy for that. This is suboptimal, it pollutes the working copy and prevents them to keep the file around in cache. We do not want to use the cache directory for the share target, it might be on a different file system. So instead, we (try to) create the directory if it is missing. This is a simple change that fixes the current behavior regression on stable. On default, we should probably ensure the proper directories are created when initializing the repository. We should also introduce a 'wcache' directory to hold cache file related to the working copy. This would clarify the cache situation regarding shares. The tests catch a couple of other affected cases.

graft: do not try to skip rev derived from ancestor more than once (issue6024) We check 'x in revs' in other cases, so let's do the same. The test case credits to Tom Prince.

subrepo: print the status line before creating the peer for better diagnostics I ran into a problem where I tried updating to a different branch, and the process appeared to hang. It turned out that the subrepo revision wasn't available locally, and I must have originally cloned it from an `hg serve -S` on a machine that currently wasn't serving anything. It took 2+ minutes to timeout, and didn't mention what it was connecting to even then. There are a couple of other issues in this scenario too. - The repo is dirty after the failed checkout because the top level repo is updated first. We should probably make 2 passes- top down to pull everything needed, and then do an update once everything is in place. - Something must be reading .hgsubstate from wdir because if the same merge command is run after the timeout, a prompt is issued that the local and remote subrepo diverged, instead of hanging. But it lists the local version and remote version as having the same hash.

tests: allow for 100% of profiled time in sleep in test-profile.t I'm getting an annoying failure in this test on our builder, and I *think* what's happening is that the profiler is taking _just_ long enough to start that we're spending 100% of the profiled time in the sleep function, which was causing the leading space to not be printed since the 100 was in the first column of output. Differential Revision: https://phab.mercurial-scm.org/D5272

copystore: provide unit to ui.makeprogress()

verify: provide unit to ui.makeprogress()

tests: fix wireproto redirection test on systems without tls1.2 Our automated package builder has some ancient configuration that lacks modern TLS, which is how we noticed this. Tested: the test now passes on both macOS High Sierra (has tls1.2) and Ubuntu Trusty (which does not).

phabricator: ensure the command summaries are available in extension help Previously, `hg help phabricator` listed the 3 supported commands at the bottom of the extension help, but said "no help text available".

hgweb: cast bytearray to bytes PEP-3333 seems to indicate that bytes is the only allowed type that can be used to express the output of a WSGI application. And some WSGI environments seem to enforce this (mod_wsgi does). This commit universally casts bytearray instances to bytes to appease the WSGI specification. I found this because wireprotov2 is emitting bytearray instances. I'd like to keep things that way because the way it builds a data structure, bytearray is more efficient. I'd rather keep the low-level code efficient (and using bytearray) and cast at the edges than impose a performance penalty on code that may run outside WSGI contexts.

help: unjumble the list of default config values for `internals.config`

tweakdefaults: remove commands.resolve.mark-check=abort, it is too broken See issue6020 for the current case. I don't want to continue attempting to fix this on the stable branch, so I'm removing from tweakdefaults and will send fixes meant for the default branch and 4.9. Differential Revision: https://phab.mercurial-scm.org/D5225

resolve: when resolve.mark-check=abort, downgrade to warning if pats specified Previously, with --config resolve.mark-check=abort, running `hg resolve -m foo` would abort and emit a message saying to use --all. This command does not work, though: `hg resolve -m foo --all`, and it's really weird for --all to be the "--force" flag. My original goal with the option was to make it so that `hg resolve -m` (no filename arguments) was safer, which is why --all is used; in my mind, `hg resolve -m foo` should always mark it as resolved, and `--all` is how you specify "all the files", so that's why I chose `hg resolve -m --all` as the way out of `hg resolve -m` aborting. This commit makes all of this work the way it was meant to in my head :) Differential Revision: https://phab.mercurial-scm.org/D5218

Added signature for changeset a91a2837150b

Added tag 4.8 for changeset a91a2837150b

rust: fix signature of rustlazyancestors_init() function Obviously, sizeof(int) != mem::size_of::<usize>() on amd64, though the argument would be passed in 64-bit register anyway.

tests: require SQLite 3.8.3+ as sqlitestore relies on "WITH" clause The test fails on gcc112 because the SQLite is too old. https://sqlite.org/changes.html#version_3_8_3

relnotes: various tweaks for release notes Stop filtering out commits that are expected to be covered by releasenotes extension: now we want two lists, one for WhatsNew and one for ReleaseX.Y. Use `only(stoprev, startrev)` to make `relnotes -h` output be actually true about what revisions are included. More filter rules, mostly obvious. More classifying rules to have less things in "unsorted". Looks like nargs=1 was just making args.startrev and args.stoprev be lists for no reason. BC and API sections are renamed to what we're using on the WhatsNew page, and also just skipped if empty.

delta: skip "empty delta" optimisation for non-general case (issue6006) Non-general delta repository cannot delta against anything than prev. So even if the delta to prev is empty we should use it. This is similar to the change made in bafa1c4bb7a8. Differential Revision: https://phab.mercurial-scm.org/D5201

narrow: fix copies._fullcopytracing() narrowspec filtering in graft case I broke this too in 707c3804e607 (narrow: move copies overrides to core, 2018-09-28). Hopefully I'm done fixing things broken by that commit now. Differential Revision: https://phab.mercurial-scm.org/D5213

tests: demonstrate broken copies._fullcopytracing() Turns out copies._fullcopytracing() was also broken. Differential Revision: https://phab.mercurial-scm.org/D5212

narrow: make copies.pathcopies() filter with narrowspec again I broke this in 707c3804e607 (narrow: move copies overrides to core, 2018-09-28). Differential Revision: https://phab.mercurial-scm.org/D5203

tests: demonstrate broken copies.pathcopies() Differential Revision: https://phab.mercurial-scm.org/D5202

setup: explain to distutils how we write rc versions When we use a rc version number (e.g. 4.8rc0), bdist_msi is using distutils.StrictVersion to parse it into a tuple of numbers. By default, StrictVersion.version_re only recognizes [ab] for alpha/beta, where mercurial may use '-rc' or 'rc'. This change makes StrictVersion parse correctly our version numbers, so that bdist_msi doesn't fail on rc versions.

changegroup: restore default node ordering (issue6001) Changeset db5501d9 changed the default node ordering from "storage" to "linearize". While the new API is more explicit and cleaner, the "linearize" order is problematic on certain repositories like netbeans where it makes bundling slower the more nodes we bundle. Pushing and pulling 100 changesets was ~20% slower and pushing and pulling 1000 changesets was ~600% slower. A very quick analysis of profile traces showed that the pull operation was taking more time creating the delta. Putting back the old default order seems to be the safe option. With more time during the next cycle, we can understand better the impact of sorting with the DAG order by default, the source of the regression and how to mitigate it. /!\ We are still waiting for the full performance impact but with this patch, bundling and pulling locally (not on the performance workstation) 1000 changesets on the netbeans repository is as fast as before the regression. Differential Revision: https://phab.mercurial-scm.org/D5196

changegroup: introduce an explicit linear sorting We still need to linearize the revisions in some cases, introduce an explicit `linear` sorting before changing back the default order. Differential Revision: https://phab.mercurial-scm.org/D5195

fix: disable use of thread-based worker getfixes() accesses to repo, changectx, filectx, etc., so I believe there are code paths triggering data race. Mercurial API isn't thread safe in general.

configitems: rename the config to prevent adding an alias in future Right now the config option looks like: [experimental.server] stream-narrow-clones= which does not match how config options are generally defined in core. So let's rename this to: [experimental] server.stream-narrow-clones= before the new release so that we don't have to add an alias in future for this. Differential Revision: https://phab.mercurial-scm.org/D5198

sparse-revlog: only refine delta candidates in the sparse case (issue6006) Starting with 5aef5afa8654, a valid delta parent might be "refined". This allows repository using sparse-revlog to produce better delta chain by using better intermediate snapshot base. However, this refining step was performed in all cases, including for repository not using sparse-revlog. This could produce a strange chain in the general delta case and corrupted repository in the non-general delta case. We now skip this step unless sparse-revlog is in use. In issue 6006, Yuya Nishihara provided a test case using an external repository, so we did not include it. Finding "laboratory" condition to reproduce this case and implementing an efficient test reproducing it is a bit tricky. We do not foresee to have the time to provide one by the release date. Differential Revision: https://phab.mercurial-scm.org/D5197

http: work around custom http client classes that refuse extra attrs I have no idea what is going on with our custom http client code at Google, but it chokes on these extra attributes we're tucking on http clients. Since it feels more than a little wrong to just stuff extra data on a client, let's degrade gracefully when the client class refuses the attributes.

crecord: make nextsametype() check that parent item exists (issue6009) Items that represent files in curses interface don't have parents.

help: describe what ui.tweakdefaults changes, concretely Currently, one has to look at the code. A couple things are suboptimal: - probably not translatable - lines don't get wrapped (a couple are a bit too long) but it seems to better this way than without help at all. Differential Revision: https://phab.mercurial-scm.org/D5187

logexchange: convert paths to unix when detecting the active path This fixes the problem in the tests[1] where Windows was showing the whole path as the remotename for local repositories. Somebody with a better understanding of this extension should probably take a deeper look. There may be other cases that need to be converted- specifically the `elif not instance` and the missing `else` cases in activepath(). I also noticed when adding debug prints that the absolute path is stored in the file, probably not normalized. (It's wrapped up in $TESTTMP.) [1] https://buildbot.mercurial-scm.org/builders/Win7%20x86_64%20hg%20tests/builds/1042/steps/run-tests.py%20%28python%202.7.13%29/logs/stdio

help: update the default value specified for `profiling.time-track` I tried conditionalizing this in a `.. container::` block, but that seemed to add an extra blank line between the main text and the parenthetical.

profiling: revert the default mode back to 'cpu' on Windows On Windows, os.times() only returns user and system times. Real elapsed time is 0. That results in no actual times reported, an end wall time of 0.000000, and seemingly randomly sorted stack frames. This at least provides test stability in test-profile.t. I kind of think that `default=pycompat.iswindows and 'cpu' or 'real'` would be a better way to set the default in configitems, but I didn't see any other examples of this, and thought maybe there's a reason for that. That might allow plugging the value into the help text automatically- the documented default wasn't updated in db0dba2d157d.

phase: add an archived phase This phase allows for hidden changesets in the "user space". It differs from the "internal" phase which is intended for internal by-product only. There have been discussions at the 4.8 sprint to use such phase to speedup cleanup after history rewriting operation. Shipping it in the same release as the 'internal-phase' groups the associated `requires` entry. The important bit is to have support for this phase in the earliest version of mercurial possible. Adding the UI to manipulate this new phase later seems fine. The current plan for archived usage and user interface are as follow. On a repository with internal-phase on and evolution off: * history rewriting command set rewritten changeset in the archived phase. (This mean updating the cleanupnodes method). * keep `hg unbundle .hg/strip-backup/X.hg` as a way to restore changeset for now (backup bundle need to contains phase data) * [maybe] add a `hg strip --soft` advance flag (a light way to expose the feature without getting in the way of a better UI) Mercurial 4.8 freeze is too close to get the above in by then. We don't introduce a new repository `requirement` as we reuse the one introduced with the 'archived' phase during the 4.8 cycle.

exewrapper: apply clang-format to silence test-check-clang-format.t

Added signature for changeset 956ec6f1320d

Added tag 4.8rc0 for changeset 956ec6f1320d

merge to stable for 4.8 release freeze

shortest: never emit 0-length prefix even if unique It turned out that the pure version of our code for finding the shortest unique nodeid prefix would return a 0-length string if that was unique (because there was at most one revision in the disambiguation set). That's kind of correct, but it can't be used as input, so we shouldn't return it. Let's just adjust the given minlength up to at least 1. This fixes test-template-functions.t, which was failing in pure mode. Differential Revision: https://phab.mercurial-scm.org/D5181

logtoprocess: sends the canonical command name to the subprocess One of the use-case of logtoprocess is to monitor command duration. With the current code, we only get whatever command name the user typed (either abbreviated or aliased). This makes analytics on the collected data more difficult. Stores the canonical command name in the request object. Pass the stored canonical name in the `req.ui.log("commandfinish", ...)` call as keyword argument to not break potential string formatting. Pass the value as the environment variable named `LTP_COMMAND` to the called script. Differential Revision: https://phab.mercurial-scm.org/D4820

logtoprocess: fix message formatting The logtoprocess used to try formatting the message using keyword options instead of always using the rest of the arguments. Update it to match blackbox behavior. Differential Revision: https://phab.mercurial-scm.org/D5180

profiling: move default mode to "real" time Mercurial operations involve a lot of disks or network access. These impact command runtime significantly and it seems important to report them in our default profiling output. Having the right default means that we don't forget them when asking people to produces profiling traces or when doing profiling ourselves. Moving to "real time" by default will remove the need to think about activating it on most occasions. The "CPU" time-based profiling is still accessible when necessary.

streamclone: abort when client needs to handle obsmarkers, but doesn't When client doesn't have any of obsolescence markers exchange capabilities, then it's safe to say it can't handle obsmarkers. However, if it understands even one format version, then stream clones are fine -- client can use "obsmarkers" bundle2 part.

streamclone: include obsstore file into stream bundle if client can read it

setup: build exewrapper with Unicode support on py3 I didn't see a compiler switch documented anywhere, but I diffed the command line for full VC++ project when toggling between MBCS and Unicode. This is all they do.

exewrapper: convert to _tcsxxx functions for Unicode compatability This fixes more than 50 tests on py3 on Windows when enabled, mostly hooks and such that invoked `hg` directly. 187 left to go. I skipped doing the abort printing with Unicode because of apparent issues with MinGW [1]. It may be moot though, as MinGW isn't listed as a supported compiler after 3.4 [2]. [1] https://stackoverflow.com/questions/17700797/printf-wprintf-s-s-ls-char-and-wchar-errors-not-announced-by-a-compil [2] https://wiki.python.org/moin/WindowsCompilers

exewrapper: drop an unused variable

commands: restore compatibility for "^cmd" registration (issue6005) This is done at loading time, where ui is available.

exchangev2: support fetching shallow files history This commit teaches the exchangev2 client code to handle fetching shallow files data. Only shallow fetching of files data is supported: shallow fetching of changeset and manifest data is explicitly not yet supported. Previously, we would fetch file revisions for changesets that were received by the current pull operation. In the new model, we calculate the set of "relevant" changesets given the pull depth and only fetch files data for those changesets. We also teach the "filesdata" command invocation to vary parameters as needed. The implementation here is far from complete or optimal. Subsequent pulls will end up re-fetching a lot of files data. But the application of this data should mostly be a no-op on the client, so it isn't a big deal. Depending on the order file revisions are fetched in, revisions could get inserted with the wrong revision number relationships. I think the best way to deal with this is to remove revision numbers from storage and to either dynamically derive them (by reconstructing a DAG from nodes/parents) or remove revision numbers from the file storage interface completely. A missing API that we'll likely want to write pretty soon is "ensure files for revision(s) are present." We can kind of cajole exchangev2.pull() to do this. But it isn't very efficient. For example, in simple cases like widening the store to obtain data for a single revision, it is probably more efficient to walk the manifest and find exactly which file revisions are missing and to make explicit requests for just their data. In more advanced cases, asking the server for all files data may be more efficient, even though it requires sending data the client already has. There is tons of room for future experimentation here. And TBH I'm not sure what the final state will be. Anyway, this commit gets us pretty close to being able to have shallow and narrow checkouts with exchangev2/sqlite storage. Close enough that a minimal extension should be able to provide fill in the gaps until the code in core stabilizes and there is a user-facing way to trigger the narrow/shallow bits from `hg clone` without also implying using of the narrow extension... Differential Revision: https://phab.mercurial-scm.org/D5169

sqlitestore: support for storing revisions without their parents This commit kinda/sorta implements the equivalent of ellipsis nodes for the SQLite storage backend. Without implementing full blown ellipsis nodes (and the necessary support for them in the wire protocol), we instead teach the store to rewrite the p1 and p2 nodes to nullid when the incoming parent isn't in the local store. This allows servers to remain dumb and send the real parent and have the clients deal with the missing parent problem. This obviously isn't ideal because a benefit of ellipsis nodes is we can insert a fake parent to ellide missing changesets. But neither solution is ideal because it drops the original parent from storage. We could probably teach the SQLite store to retain the original parent and handle missing parents at read time. However, parent revisions are stored as integers and it isn't trivial to store an "empty" revision in the store yet, which would be necessary to represent the "missing" parent. The store is somewhat intelligent in trying to remove the missing parents metadata when the revision is re-added. But, revision numbers will be all messed up in that case, so I'm not sure it is worth it. At some point we'll likely want to remove the concept of revision numbers from the database and have the store invent them at index generation time. Or even better, we can do away with revision numbers from the file storage interface completely. We'll get there eventually... Differential Revision: https://phab.mercurial-scm.org/D5168

wireprotov2: support exposing linknode of file revisions When supporting shallow file storage, clients may fetch file revisions by changeset. But they may not readily know which changeset introduced a specific file revision. The "linknode" is used to record which changeset introduces which file revision. This commit teaches the "filedata" and "filesdata" wire protocol commands to expose the linknode for file revisions. The implementation is likely wrong when hidden changesets are in play, since the linknode may refer to a hidden changeset. We can deal with this problem later. Differential Revision: https://phab.mercurial-scm.org/D5167

localrepo: support marking repos as having shallow file storage Various operations against repositories need to know if repository storage is full or partial. For example, a checkout (including possibly a widening of a sparse checkout), needs to know if it can assume all file revisions are available or whether to look for missing revisions first. This commit lays the plumbing for doing that. We define a repo creation option that indicates that shallow file storage is desired. The SQLite store uses this creation option to add an extra repo requirement indicating file storage is shallow. A new repository feature has been added to indicate that file storage is shallow. The SQLite store adds this feature when the shallow file store requirement is present. Code can now look at repo.features to determine if repo file storage may be shallow and take additional actions if so. While we're here, we also teach the SQLite store to handle the narrow repo requirement, which gets added when making narrow clones. Differential Revision: https://phab.mercurial-scm.org/D5166

repository: teach addgroup() to receive data with missing parents The way the narrow extension works today, the server rewrites outgoing changegroup data to lie about parents when the parents data is missing. It adds the ellipsis flag to the revision so it can be recorded as such in the revlog. In the new wire protocol, such rewriting does not occur on the server (at least not yet anyway). Instead, it is up to the client to recognize when it has received a revision without its parents. This means rewriting will be performed on the client. Furthermore, the mechanism for storing a shallow revision may differ from store to store. For example, the revlog store uses the ellipsis flag to denote a revision's parents have been rewritten. But a non-revlog store may wish to store things differently. And, some stores may not even support receiving shallow revision data! Therefore, it makes sense for the store itself to be making decisions about what to do when they receive revision data without their parents. This commit teaches the addgroup() bulk insert method to accept a boolean argument that indicates whether the incoming data may lack parent revisions. This flag can be set when receiving "shallow" data from a remote. The revlog implementation of this method has been taught to rewrite the missing parent(s) to nullid and to add the ellipsis flag to the revision when a missing parent is encountered. But it only does this if ellipsis flags are enabled on the repo and the incoming data is marked as possibly shallow. An error occurs otherwise. Differential Revision: https://phab.mercurial-scm.org/D5165

commands: support passing depth to hg.clone() This will allow extensions to add --depth or other arguments to control depth fetching. Differential Revision: https://phab.mercurial-scm.org/D5164

filelog: add a hasnode() method (API) Missing in the file storage interface is the ability to query whether a specified value is a known node. This commit defines that interface member and implements it on the revlog and sqlite file stores. Storage unit tests have been added. The revlog implementation is a bit more complicated because index lookups don't consistently raise the same exception. For SQLite, we can simply look for a key in a dict. Differential Revision: https://phab.mercurial-scm.org/D5163

lfs: consult the narrow matcher when extracting pointers from ctx (issue5794) I added a testcase for lfs to all narrow tests, and the following failed: test-narrow-acl.t test-narrow-exchange.t test-narrow-patterns.t test-narrow-strip.t test-narrow-trackedcmd.t test-narrow-widen.t test-narrow.t The first two still have errors in the pretxnchangegroup on clone and (receiving a) push, which I'm still looking into (4d63f3bc1e1a fixed something in this area already). These two modified tests seem to cover the things that failed in the remaining narrow tests, i.e. `hg tracked` and `hg strip`, so I didn't bother enabling the testcases elsewhere. Maybe we should, but it's 68 tests total.

statprof: fix overflow while skipping boilerplate parts I got IndexError randomly because of stack[i] where i = len(stack).

statprof: fix indent level of fp.write() (issue6004) It was changed at 9d3034348c4f by mistake.

py3: stringify setupversion on Windows This was stringified a few lines above for non Windows platforms, but `version` remains bytes. The old code effectively undid the conversion, and triggered a warning in setuptools when building.

tests: add coverage for some untested areas of hgweb The fact that these mimetype guesses weren't blowing up anywhere on py3 prior to 9310037f0636 was the giveaway. The annotate function is a bit unusual in that it renders the page with a 500 in the middle, so I left the HTML output. For the other functions, checking the access log is enough.

statprof: update the name as the i increases (issue6003) 2864f8d3fcd6 while working on py3 fix, take out the name building out of the loop so we were not building the new stack-name for each i, rather we were using the first one again and again. The test changes shows the profile is now working. Differential Revision: https://phab.mercurial-scm.org/D5172

test: show more profile lines in test-profile.t This shows that we don't output anything after the first line and demonstrate issue6003. Differential Revision: https://phab.mercurial-scm.org/D5171

keepalive: use getattr to avoid AttributeErrors when vcr is in use Fixes test-phabricator.t. Differential Revision: https://phab.mercurial-scm.org/D5160

phabricator: do more of the VCR work in demandimport.deactivated() If I don't do this, VCR gets confused looking for pycurl and other libraries. I have no idea how this ever worked. Differential Revision: https://phab.mercurial-scm.org/D5159

tests: sleep longer in test-logtoprocess.t We should probably write some sort of helper that can wait N seconds for all specified values to appear in a file or something, but for now this will fix the FreeBSD buildbot. Differential Revision: https://phab.mercurial-scm.org/D5157

tests: fix pyflakes warning in test-duplicateoptions.py Differential Revision: https://phab.mercurial-scm.org/D5158

branchmap: avoid changelog and attribute lookups in replacecache() This should make things faster. I'm not sure which operations would benefit from it though. Maybe branchmap application on clone? Differential Revision: https://phab.mercurial-scm.org/D5162

branchmap: pass changelog into branchmap functions As part of building the branchmap, we loop over revs and call branchmap() or _branchmap(). Previously, these functions were accessing repo.changelog. We know from past experience that repo.changelog in loops is bad for performance. This commit teaches the branchmap code to pass a changelog instance into branchmap() and _branchmap() so we don't need to pay this penalty. On my MBP, this appears to show a speedup on a clone of the mozilla-unified repo: $ hg perfbranchmap --clear-revbranch ! base ! wall 21.078160 comb 21.070000 user 20.920000 sys 0.150000 (best of 3) ! wall 20.574682 comb 20.560000 user 20.400000 sys 0.160000 (best of 3) $ hg perfbranchmap ! base ! wall 4.880413 comb 4.870000 user 4.860000 sys 0.010000 (best of 3) ! wall 4.573968 comb 4.560000 user 4.550000 sys 0.010000 (best of 3) Differential Revision: https://phab.mercurial-scm.org/D5161

fuzz: move many initialization steps into LLVMFuzzerInitialize Doing this means that things we intentionally leak (eg type objects) no longer confuse AddressSanitizer, so now we can run the fuzzer MUCH longer. Differential Revision: https://phab.mercurial-scm.org/D5154

bundle2: fix broken compression engine assertion bundletype() is a function, so it needs to be called, and it is documented to return a 2-tuple. This code is untested, so that's why we haven't noticed the bad assertion. Differential Revision: https://phab.mercurial-scm.org/D5155

tests: glob over a difference between Windows 7 and Window 10 The error value is 11001 on Windows 10. I have no idea why it changed, but it seems unimportant.

py3: fix module imports in test-highlight.t The hash changes are because the *.py file is committed to the repo.

py3: fix module imports in tests, as flagged by test-check-module-imports.t I have no idea why these aren't flagged with python2. I excluded test-highlight.t for now to make this easier to review- the changed code is committed to a repo, which has cascading changes on the rest of the test. There's a mix of bytes and str in the imports dict of contrib/import-checker.py that crashed it half way through listing out these errors. I couldn't figure out how to fix that properly, so I was lazy and applied this on py3, to find the rest of the errors: diff --git a/contrib/import-checker.py b/contrib/import-checker.py --- a/contrib/import-checker.py +++ b/contrib/import-checker.py @@ -626,7 +626,12 @@ def find_cycles(imports): top.foo -> top.qux -> top.foo """ cycles = set() - for mod in sorted(imports.keys()): + def sort(v): + if isinstance(v, bytes): + return v.decode('ascii') + return v + + for mod in sorted(imports.keys(), key=sort): try: checkmod(mod, imports) except CircularImport as e:

lfs: don't add extension to hgrc after conversion (BC) This is in the spirit of bcf72d7b1524.

addremove: add "ui." prefix to message color keys I don't like fully-colorized status/warning messages, and I want to disable them at all. If we'd supported a syntax like 'color.ui.*=none', I could easily turn addremove.added/removed off as well as ui.error. This patch is just for that. Since addremove colors aren't released yet, which were added at ddc1da134772, there are no compatibility concerns.

update: clarify update() call sites by specifying argument names merge.update() takes a lot of parameters and I get confused all the time which is which. Differential Revision: https://phab.mercurial-scm.org/D5153

debugcommands: avoid stack trace from debugindexstats in pure mode This has been broken since I added it in d71e0ba34d9b (debugcommands: add a debugindexstats command, 2018-08-08). This patch also fixes the test. Differential Revision: https://phab.mercurial-scm.org/D5152

tests: fix up pure case of test-sqlitestore.t This is clearly what the line should read based on the "force to zlib" section below, so I'm guessing it just got overlooked during development. Differential Revision: https://phab.mercurial-scm.org/D5151

tests: don't emit false failures when sqlite3 is missing I'm honestly surprised we have buildbot coverage for this, but we do! Differential Revision: https://phab.mercurial-scm.org/D5150

py3: get around IOError variants in test-commandserver.t

py3: don't use traceback.print_exc() in commandserver.py It doesn't support a bytes stream on Python 3. This makes a traceback being sent by one frame, but that shouldn't matter.

py3: invalidate repository cache with system-string keys # skip-blame just a few r'' prefixes

py3: system-stringify file mode in commandserver.py # skip-blame just r'' prefixes

py3: alias next to __next__ in commandserver.py

py3: system-stringify list of attributes to be forwarded from commandserver.py # skip-blame just some r'' prefixes

py3: import StringIO from test utility to test-commandserver.t

py3: use bprint() helper in test-commandserver.t

py3: byte-stringify most literals in test-commandserver.t print() calls will be replaced by bprint(). # skip-blame just tons of b'' prefixes.

localrepo: ensure we properly %-format int in exception throw I'm not thrilled with this, but it'll do. Differential Revision: https://phab.mercurial-scm.org/D5107

py3: add a r'' prefix in mercurial/exchange.py # skip-blame because just r'' prefix This fixes test-narrow-acl.t on py3 which was broken by one of the earlier patches. Differential Revision: https://phab.mercurial-scm.org/D5149

py3: add 5 new passing tests to whitelist caught by buildbot Thanks to everyone who is putting efforts in making hg py3 compatible. Differential Revision: https://phab.mercurial-scm.org/D5148

py3: fix test-import-context.t

py3: restore perfstartup() prior to b456b2e0ad9f on Windows Otherwise the test errors out with: --- c:/Users/Matt/projects/hg_py3/tests/test-contrib-perf.t +++ c:/Users/Matt/projects/hg_py3/tests/test-contrib-perf.t.err @@ -184,6 +184,8 @@ $ hg perfrevrange $ hg perfrevset 'all()' $ hg perfstartup + 'b'c:' is not recognized as an internal or external command, + operable program or batch file. $ hg perfstatus $ hg perftags $ hg perftemplating

help: document the server capabilities added by the LFS extension I didn't bother marking these experimental because it references the extension that is already marked experimental.

py3: fix test-propertycache.py on Windows

commands: adjust metavariables as appropriate Apart from looking better in hg help command, these strings are also helpful when generating shell completions programmatically.

match: fix up a repr to not crash on Python 3 Differential Revision: https://phab.mercurial-scm.org/D5120

narrow: when widening, don't include manifests the client already has When widening, we already don't include the changelog (since f1844a10ee19) and files that the client already has (since c73c7653dfb9). However, we still include all manifests needed for the new narrowspec. When using flat manifests, that means we resend all the manifests even though the client necessarily has all of them. For tree manifests, we unnecessarily resend the root manifests and any subdirectory manifests that the client already has. This patch makes it so we no longer resend manifests that the client already has. It does so by passing an extra matcher to the changegroup packer and it uses that for filtering out directories matching the old matcher's visitdir(). For consistency between directories and files, it also makes the filtering of files look at both old and new matcher rather than passing in a diff matcher as we did before. Differential Revision: https://phab.mercurial-scm.org/D4895

tests: add test for widening from an empty clone Narrow clones that track no paths currently don't even include the root manifest (which is the only manifest when using flat manifests). That means that when we widen from such a clone, we need to make sure that we send the root manifest (and other manifests if using tree manifests). That currently works because we always resend all manifest that match the new narrowspec. However, we're about to stop resending manifests that the client already has and there's a risk of this breaking then, so let's add a test. Differential Revision: https://phab.mercurial-scm.org/D5143

subrepo: access status members by name instead of by position Taking my first Mercurial project closer to completion. Differential Revision: https://phab.mercurial-scm.org/D5144

revisions: when using prefixhexnode, ensure we prefix "0" Previously, if using `experimental.revisions.disambiguatewithin` (and it didn't include rev0), and '0' was the shortest identifier in that disambiguation set, we printed it as the shortest *without* a prefix. This was because we had logic to determine "if the prefix is a pure integer, but starts with 0, we don't need to prefix with 'x': 01 is not a synonym for revision #1", but didn't handle the case where prefix == 0 (which is a pure integer, and starts with 0... but it *is* "rev0"). Differential Revision: https://phab.mercurial-scm.org/D5113

store: pass matcher to store.datafiles() To get narrow stream clones working, we need a way to filter the storage files using a matcher. This patch adds matcher as an argument to store.walk() and store.datafiles() so that we can filter the files returned according to the matcher. Differential Revision: https://phab.mercurial-scm.org/D4850

streamclone: pass narrowing related info in _walkstreamfiles() This patch build a matcher using the include and exclude arguments we have in generatev2() and pass that matcher into _walkstreamfiles(). This will help us in filtering files we stream depending on the includes and excludes passed in by the user. Differential Revision: https://phab.mercurial-scm.org/D4851

streamclone: new server config and some API changes for narrow stream clones This patch introduces a new server config `experimental.server.stream-narrow-clones` which if set to True will advertise that the server supports narrow stream clones. This patch also pass on the includes and excludes from getbundle command to streamclone generation code. There is a test added to show that the includepats and excludepats are correctly passed. Upcoming patches will implement storage layer filtering for streamclones and then we can remove the temporary error and plug in the whole logic together to make narrow stream clones working. Differential Revision: https://phab.mercurial-scm.org/D5137

narrow: only send the narrowspecs back if ACL in play I am unable to think why we need to send narrowspecs back from the server. The current state adds a 'narrow:spec' part to each changegroup which is generated when narrow extension is enabled. So we are sending narrowspecs on pull also. There is a problem with sending the narrowspecs the way we are doing it right now. We add include and exclude as parameter of the 'narrow:spec' bundle2 part. The the len of include or exclude string increase 255 which is obvious while working on large repos, bundle2 generation code breaks. For more on that refer issue5952 on bugzilla. I was thinking why we need to send the narrowspecs back, and deleted the 'narrow:spec' bundle2 part generation code and found that only narrow-acl test has some failure. With this patch, we will only send the 'narrow:spec' bundle2 part if ACL is enabled because the original narrowspecs in those cases can be a subset of narrowspecs user requested. There are phase related output change in couple of tests. The output change shows that we are now dealing in public phases completely. So maybe sending the narrow:spec bundle2 part was preventing phases being exchanged or phase bundle2 data being applied. Differential Revision: https://phab.mercurial-scm.org/D4931

zsh_completion: add -l/--list flag for hg bookmarks completion Flags in parentheses are mutually exclusive. Logic is taken from commands.py: selactions = [k for k in ['delete', 'rename', 'list'] if opts.get(k)] if len(selactions) > 1: raise error.Abort(_('--%s and --%s are incompatible') % tuple(selactions[:2])) ... if rev and action in {'delete', 'rename', 'list'}: raise error.Abort(_("--rev is incompatible with --%s") % action) if inactive and action in {'delete', 'list'}: raise error.Abort(_("--inactive is incompatible with --%s") % action) Differential Revision: https://phab.mercurial-scm.org/D5142

zsh_completion: fix a couple of flags still not being perfect Differential Revision: https://phab.mercurial-scm.org/D5141

zsh_completion: use $_hg_remote_opts after it is defined Before this patch, zsh wouldn't complete --ssh, --remotecmd or --insecure for hg clone. While at it, replace --uncompressed by --stream. Differential Revision: https://phab.mercurial-scm.org/D5140

tests: fix "running x tests using y ... " output in a few more places These seem to have been missed by 1039404c5e1d (run-tests: print number of tests and parallel process count, 2018-10-13). Differential Revision: https://phab.mercurial-scm.org/D5145

py3: fix test-hardlinks.t Differential Revision: https://phab.mercurial-scm.org/D5096

exchange: support declaring pull depth Upcoming commits will teach exchangev2 how to perform a shallow clone. This commit teaches hg.clone(), exchange.pull(), and exchange.pulloperation to recognize a request for a shallow clone by having the caller specify a numeric depth of the maximum number of ancestor changesets to fetch. There are certainly other ways we could control shallow-ness. But this one is simple to implement and is also how the narrow extension controls things. So it seems to make sense to start here. Differential Revision: https://phab.mercurial-scm.org/D5136

exchangev2: support for calling rawstorefiledata to retrieve raw files This is somewhat hacky. For that I apologize. At the 4.8 Sprint, we decided we wanted to land support in wireprotov2 for doing a partial clone with changelog and manifestlog bootstrapped from a "stream clone" like primitive. This commit implements the client-side bits necessary to facilitate that. If the new server-side command for obtaining raw files data is available, we call it to get the raw files for the changelog and manifestlog. Then we fall through to an incremental pull. But when fetching files data, instead of using the list of a changesets and manifests that we fetched via the "changesetdata" command, we do a linear scan of the repo and resolve the changeset and manifest nodes along with the manifest linkrevs. Differential Revision: https://phab.mercurial-scm.org/D5135

wireprotov2: implement command for retrieving raw store files Implementing shallow clone of the changelog is hard. We want the 4.8 release to have a fast implementation of partial clone in wireprotov2. In order to achieve fast, we can't use deltas for transferring changelog and manifestlog data. Per discussions at the 4.8 sprint, this commit implements a somwwhat hacky and likely-to-be-changed-drastically-or-dropped command in wireprotov2 that facilitates access to raw store files, namely the changelog and manifestlog. Using this command, clients can perform a "stream clone" of sorts for just the changelog and manifestlog. This will allow clients to fetch the changelog and manifest revlogs, stream them to disk (which should be fast), then follow up filesdata requests for files revision data for a particular changeset. Differential Revision: https://phab.mercurial-scm.org/D5134

wireprotov2: add response type that serializes to indefinite length bytestring This will be needed in a future patch. Differential Revision: https://phab.mercurial-scm.org/D5133

exchangev2: recognize narrow patterns when pulling pulloperation instances were recently taught to record file include and exclude patterns to facilitate narrow file transfer. Teaching the exchangev2 code to transfer a subset of files is as simple as constructing a narrow matcher from these patterns and filtering all seen file paths through it. Keep in mind that this change only influences file data: we're still fetching all changeset and manifest data. So, there's still a ton of "partial clone" to implement in exchangev2. On a personal note, I derive gratification that this feature requires very few lines of new code to implement. To test this, we implemented a minimal extension which allows us to specify --include/--exclude to clone. While the narrow extension provides these arguments, I explicitly wanted to test this functionality without the narrow extension enabled, as that extension monkeypatches various things and I want to isolate the behavior of core Mercurial. Differential Revision: https://phab.mercurial-scm.org/D5132

sqlitestore: file storage backend using SQLite This commit provides an extension which uses SQLite to store file data (as opposed to revlogs). As the inline documentation describes, there are still several aspects to the extension that are incomplete. But it's a start. The extension does support basic clone, checkout, and commit workflows, which makes it suitable for simple use cases. One notable missing feature is support for "bundlerepos." This is probably responsible for the most test failures when the extension is activated as part of the test suite. All revision data is stored in SQLite. Data is stored as zstd compressed chunks (default if zstd is available), zlib compressed chunks (default if zstd is not available), or raw chunks (if configured or if a compressed delta is not smaller than the raw delta). This makes things very similar to revlogs. Unlike revlogs, the extension doesn't yet enforce a limit on delta chain length. This is an obvious limitation and should be addressed. This is somewhat mitigated by the use of zstd, which is much faster than zlib to decompress. There is a dedicated table for storing deltas. Deltas are stored by the SHA-1 hash of their uncompressed content. The "fileindex" table has columns that reference the delta for each revision and the base delta that delta should be applied against. A recursive SQL query is used to resolve the delta chain along with the delta data. By storing deltas by hash, we are able to de-duplicate delta storage! With revlogs, the same deltas in different revlogs would result in duplicate storage of that delta. In this scheme, inserting the duplicate delta is a no-op and delta chains simply reference the existing delta. When initially implementing this extension, I did not have content-indexed deltas and deltas could be duplicated across files (just like revlogs). When I implemented content-indexed deltas, the size of the SQLite database for a full clone of mozilla-unified dropped: before: 2,554,261,504 bytes after: 2,488,754,176 bytes Surprisingly, this is still larger than the bytes size of revlog files: revlog files: 2,104,861,230 bytes du -b: 2,254,381,614 I would have expected storage to be smaller since we're not limiting delta chain length and since we're using zstd instead of zlib. I suspect the SQLite indexes and per-column overhead account for the bulk of the differences. (Keep in mind that revlog uses a 64-byte packed struct for revision index data and deltas are stored without padding. Aside from the 12 unused bytes in the 32 byte node field, revlogs are pretty efficient.) Another source of overhead is file name storage. With revlogs, file names are stored in the filesystem. But with SQLite, we need to store file names in the database. This is roughly equivalent to the size of the fncache file, which for the mozilla-unified repository is ~34MB. Since the SQLite database isn't append-only and since delta chains can reference any delta, this opens some interesting possibilities. For example, we could store deltas in reverse, such that fulltexts are stored for newer revisions and deltas are applied to reconstruct older revisions. This is likely a more optimal storage strategy for version control, as new data tends to be more frequently accessed than old data. We would obviously need wire protocol support for transferring revision data from newest to oldest. And we would probably need some kind of mechanism for "re-encoding" stores. But it should be doable. This extension is very much experimental quality. There are a handful of features that don't work. It probably isn't suitable for day-to-day use. But it could be used in limited cases (e.g. read-only checkouts like in CI). And it is also a good proving ground for alternate storage backends. As we continue to define interfaces for all things storage, it will be useful to have a viable alternate storage backend to see how things shake out in practice. test-storage.py passes on Python 2 and introduces no new test failures on Python 3. Having the storage-level unit tests has proved to be insanely useful when developing this extension. Those tests caught numerous bugs during development and I'm convinced this style of testing is the way forward for ensuring alternate storage backends work as intended. Of course, test coverage isn't close to what it needs to be. But it is a start. And what coverage we have gives me confidence that basic store functionality is implemented properly. Differential Revision: https://phab.mercurial-scm.org/D4928

storageutil: extract most of peek_censored from revlog This function is super hacky and isn't correct 100% of the time. I'm going to need this functionality on a future non-revlog store. Let's copy things to storageutil so this code only exists once. Differential Revision: https://phab.mercurial-scm.org/D5118

lfs: autoload the extension when cloning from repo with lfs enabled This is based on a patch by Gregory Szorc. I made small adjustments to clean up the messaging when the server has the extension enabled, but the client has it disabled (to prevent autoloading). Additionally, I added a second server capability to distinguish between the server having the extension enabled, and the server having LFS commits. This helps prevent unnecessary requirement propagation- the client shouldn't add a requirement that the server doesn't have, just because the server had the extension loaded. The TODO I had about advertising a capability when the server can natively serve up blobs isn't relevant anymore (we've had 2 releases that support this), so I dropped it. Currently, we lazily add the "lfs" requirement to a repo when we first encounter LFS data. Due to a pretxnchangegroup hook that looks for LFS data, this can happen at the end of clone. Now that we have more control over how repositories are created, we can do better. This commit adds a repo creation option to add the "lfs" requirement. hg.clone() sets this creation option if the remote peer is advertising lfs usage (as opposed to just support needed to push). So, what this change effectively does is have cloned repos automatically inherit the "lfs" requirement. Differential Revision: https://phab.mercurial-scm.org/D5130

testing: switch to inserting deltas As the comment in the test specifies, this was relying on storage backend implementation details. We switch to inserting a raw delta, skipping the regular insert path to ensure we have the desired outcome. This required implementing support for handling deltas in the revlog testing code. Differential Revision: https://phab.mercurial-scm.org/D5116

testing: remove expectation of error on bad node insert addgroup() doesn't necessarily validate the hashes of each incoming revision. This is an optimization that allows delta group application to complete faster. The fact that revlog raises in this particular test is an implementation detail due to the way revlogs are testing multiple deltas. Differential Revision: https://phab.mercurial-scm.org/D5115

storageutil: convert fileid to bytes to avoid cast to %s test-storage.py manages to trigger this on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5117

tests: use byte literals in test-storage.py This fixes a Python 3 breakage due to unknown key due to str/bytes type mismatch. # skip-blame just b'' literals Differential Revision: https://phab.mercurial-scm.org/D5114

py3: byte-stringify literals in test-keyword.t # skip-blame just some b'' prefixes

py3: flush std streams before/after running user code in heredoctest.py Otherwise, things written to stdout.buffer would be interleaved.

py3: rewrite StringIO fallback for Python 3

py3: reinvent print() function for contrib/hgclient.py

py3: work around unicode stdio streams in contrib/hgclient.py

py3: convert string literals to bytes in contrib/hgclient.py # skip-blame just many b'' prefixes

merge with stable

branchmap: remove redundant sort There is absoluty no benefit in sorting a list that's being merged into a set on the next line. The changelog.ancestors() call later on also doesn't benefit from a sorted sequence of revs. Differential Revision: https://phab.mercurial-scm.org/D5111

revset: drop special case of 'revset(...)' function in analyze We now have a valid no-op function. We no longer need the special case.

revset: document the `revset(...)` syntax We introduce a new "no-op" function to bear the documentation. In practice, the parsing step is skipping it so it is not even called. This will get fixed in the next changeset.

check-commit: update test expectation per removal of "double empty line" rule Follow up for 47084b5ffd80.

style: drop requirement to only use single lines between top-level objects Differential Revision: https://phab.mercurial-scm.org/D5105

py3: byteify extension in test-relink.t

f: fix a Python 3 bytes/string issue I suspect we should test this tool in isolation, but we don't yet. Oh well. Differential Revision: https://phab.mercurial-scm.org/D5061

tests: use regex instead of Python versions for archive hash changes It turns out this behavior changed between versions of Python 3. Let's just always accept either size or sha1, and move on. Differential Revision: https://phab.mercurial-scm.org/D5104

notify: a ton of encoding dancing to deal with the email module Almost fixes test-keyword.t on Python 3, but leaves us with some extremely confusing failures at the end of the test that seem related to the command server? Differential Revision: https://phab.mercurial-scm.org/D5100

tests: add missing b prefix in test-context-metadata.t # skip-blame just a b prefix Differential Revision: https://phab.mercurial-scm.org/D5109

context: raise runtime errors with sysstrs We should probably *not* use RuntimeError for this, but let's deal with that later, rather than as part of the Python 3 effort. Differential Revision: https://phab.mercurial-scm.org/D5108

rust: rustfmt config for hg-direct-ffi For now, we're duplicating it, but it would be probably a good idea to use a single one for the whole workspace (would have implications on the other crates as well)

rust: rustlazyancestors.__contains__ This changeset provides a Rust implementation of the iteration performed by lazyancestor.__contains__ It has the advantage over the Python iteration to use the 'seen' set encapsuled into the dedicated iterator (self._containsiter), rather than storing emitted items in another set (self._containsseen), and hence should reduce the memory footprint. Also, there's no need to convert intermediate emitted revisions back into Python integers. At this point, it would be tempting to implement the whole lazyancestor object in Rust, but that would lead to more C wrapping code (two objects) for little expected benefits.

help: fix a missing quote character in ui.tweakdefaults

rust: hooking into Python code We introduce a new class called 'rustlazyancestors' in the ancestors module, which is used only if parsers.rustlazyancestors does exist. The implementation of __contains__ stays unchanged, but is now backed by the Rust iterator. It would probably be a good candidate for further development, though, as it is mostly looping, and duplicates the 'seen' set. The Rust code could be further optimized, however it already gives rise to performance improvements: median timing from hg perfancestors: - on pypy: before: 0.077566s after: 0.016676s -79% - on mozilla central: before: 0.190037s after: 0.082225s -58% - on a private repository (about one million revisions): before: 0.567085s after: 0.108816s -80% - on another private repository (about 400 000 revisions): before: 1.440918s after: 0.290116s -80% median timing for hg perfbranchmap base - on pypy: before: 1.383413s after: 0.507993s -63% - on mozilla central: before: 2.821940s after: 1.258902s -55% - on a private repository (about one million revisions): before: 77.065076s after: 16.158475s -80% - on another private repository (about 401 000 revisions): before: 7.835503s after: 3.545331s -54%

py3: fix test-propertycache.py Differential Revision: https://phab.mercurial-scm.org/D5101

py3: fix test-dirstate-race.t Differential Revision: https://phab.mercurial-scm.org/D5106

help: adding a proper declaration for shortlist/basic commands (API) We previously used the '^' prefix to indicate that a command should be shown on the short list (shown for just "hg"), but that's a horrible hack, so I'm removing it. Differential Revision: https://phab.mercurial-scm.org/D5069

help: assigning topic categories Differential Revision: https://phab.mercurial-scm.org/D5068

help: assigning categories to existing commands I'm separating this into its own commit so people can bikeshed over the actual categorization (vs the support for categories). These categories are based on the help implementation we've been using internally at Google, and have had zero complaints. Differential Revision: https://phab.mercurial-scm.org/D5067

help: splitting the topics by category Differential Revision: https://phab.mercurial-scm.org/D5066

help: adding support for command categories Differential Revision: https://phab.mercurial-scm.org/D5065

notify: just use email.errors email.Errors is a proxy object to email.errors on Python 2.

rust-chg: add struct holding information needed to spawn server process The Locator will handle the initialization of the connection. It will spawn server processes as needed.

rust-chg: install logger if $CHGDEBUG is set This is modeled after the example logger and debugmsg() of chg/util.c. https://docs.rs/log/0.4.5/log/#implementing-a-logger

rust-chg: depend on log and tokio_timer I'll start porting the daemon management functions from chg of C, which will be difficult to debug without some logging facility. AFAIK, the log crate is easy-to-use and widely used. tokio_timer provides sleep() helper to be used while spawning a server process.

rust-chg: suppress panic while writing chg error to stderr Otherwise "chg >/dev/full 2>&1" would exit with 101. Spotted by test-basic.t.

logcmdutil: add a helpful assertion to catch mistyped templates early This would have made a defect in test-notify.t much easier to figure out. Differential Revision: https://phab.mercurial-scm.org/D5097

notify: adapt to new location of email module's errors Differential Revision: https://phab.mercurial-scm.org/D5099

notify: add some b prefixes # skip-blame just b prefixes Differential Revision: https://phab.mercurial-scm.org/D5098

py3: fix test-diff-color.t Differential Revision: https://phab.mercurial-scm.org/D5095

py3: fix test-revlog.t The mpatchError has a trailing comma on Python 2 but not on Python 3, so use a glob to handle both Python 2 and Python 3. Differential Revision: https://phab.mercurial-scm.org/D5093

fuzz: try *even harder* to prevent Python from looking up usernames Differential Revision: https://phab.mercurial-scm.org/D5092

wireproto: fix incorrect function name in docstring The docstring for `iwireprotocolcommandcacher` references an `onoutputfinished` method. The actual name of the function is `onfinished`. Differential Revision: https://phab.mercurial-scm.org/D5090

py3: fix test-status.t Differential Revision: https://phab.mercurial-scm.org/D5089

formatter: make debug output prettier "(glob)" won't be needed since pprintgen() can print dict items in stable order.

stringutil: allow to specify initial indent level of pprint() I want to pprint() an inner object, which starts with level=1 indent.

stringutil: make level parameter of pprintgen() 0-origin I think this makes more sense in that the level is incremented where nesting goes one more deep.

formatter: use stringutil.pprint() in debug output to drop b''

rust: exposing in parsers module To build with the Rust code, set the HGWITHRUSTEXT environment variable. At this point, it's possible to instantiate and use a rustlazyancestors object from a Python interpreter. The changes in setup.py are obviously a quick hack, just good enough to test/bench without much refactoring. We'd be happy to improve on that with help from the community. Rust bindings crate gets compiled as a static library, which in turn gets linked within 'parsers.so' With respect to the plans at https://www.mercurial-scm.org/wiki/OxidationPlan this would probably qualify as "roll our own FFI". Also, it doesn't quite meet the target of getting rid of C code, since it brings actually more, yet: - the new C code does nothing else than parsing arguments and calling Rust functions. In particular, there's no complex allocation involved. - subsequent changes could rewrite more of revlog.c, this time resulting in an overall decrease of C code and unsafety.

rust: iterator bindings to C code In this changeset, still made of Rust code only, we expose the Rust iterator for instantiation and consumption from C code. The idea is that both the index and index_get_parents() will be passed from the C extension, hence avoiding a hard link dependency to parsers.so, so that the crate can still be built and tested independently. On the other hand, parsers.so will use the symbols defined in this changeset.

rust: pure Rust lazyancestors iterator This is the first of a patch series aiming to provide an alternative implementation in the Rust programming language of the _lazyancestorsiter from the ancestor module. This iterator has been brought to our attention by the people at Octobus, as a potential good candidate for incremental "oxydation" (rewriting in Rust), because it has shown performance issues lately and it merely deals with ints (revision numbers) obtained by calling the index, whih should be directly callable from Rust code, being itself implemented as a C extension. The idea behind this series is to provide a minimal example of Rust code collaborating with existing C and Python code. To open the way to gradually rewriting more of Mercurial's Python code in Rust, without being forced to pay a large initial cost of rewriting the existing fast core into Rust. This patch does not introduce any bindings to other Mercurial code yet. Instead, it introduces the necessary abstractions to address the problem independently, and unit-test it. Since this is the first use of Rust as a Python module within Mercurial, the hg-core crate gets created within this patch. See its Cargo.toml for more details. Someone with a rustc/cargo installation may chdir into rust/hg-core and run the tests by issuing: cargo test --lib The algorithm is a bit simplified (see details in docstrings), and at its simplest becomes rather trivial, showcasing that Rust has batteries included too: BinaryHeap, the Rust analog of Python's heapq does actually all the work. The implementation can be further optimized and probably be made more idiomatic Rust.

run-tests: restore quoting the python executable for running *.py tests This was accidentally dropped in 8cf459d8b111.

tests: replace `cd ..` with an absolute path in a couple ssh tests These tests are broken under py3 on Windows to the point where the `cd ..` was actually escaping into the system wide $TEMP. The subsequent `hg init` created a repo there, and then added a local extension to the hgrc. This breaks every single subsequent test when it tries to `hg init` in its $TESTTMP, and can't load the localwrite.py extension. And since I botched this the first time and replaced the wrong `cd ..`, this just replaces all of them. I've noticed test garbage in $TEMP recently, and maybe this will help. Perhaps `hg init` shouldn't load the config for the local repo, but this is an easy enough workaround for now.

lfs: register the flag processors per repository Previously, enabling the extension for any repo in commandserver or hgweb would enable the flags on all repos. Since localrepo.resolverevlogstorevfsoptions() is called so early, the check to see if the extension is enabled on the repo (which hasn't been instantiated yet) is a bit awkward. But I don't see a better way.

revlog: allow flag processors to be applied via store options This allows flag processors to be registered to specific repos in an extension by wrapping localrepo.resolverevlogstorevfsoptions(). I wanted to add the processors via a function on localrepo, but some of the places where the processors are globally registered don't have a repository available. This makes targeting specific repos in the wrapper awkward, but still manageable.

py3: use str to query registry values on Windows This blew up launching any command if extdiff processed a tool with a regkey config.

py3: convert "usage" literal to bytes Here _() is practically an identity function, but we shouldn't pass in unicode to _().

churn: remove redundant round() To my surprise, the int() is required. Spotted by Mads when he reviewed D5063. Thanks! Differential Revision: https://phab.mercurial-scm.org/D5086

py3: use py3 as the test tag, dropping the k Differential Revision: https://phab.mercurial-scm.org/D5079

tests: fix inline extension in test-fncache.t for Python 3 # skip-blame just some b prefixing Differential Revision: https://phab.mercurial-scm.org/D5083

py3: 3 more passing tests Differential Revision: https://phab.mercurial-scm.org/D5082

simplemerge: port to Python 3 Differential Revision: https://phab.mercurial-scm.org/D5081

contrib: fix up output in check-config.py to use strs to avoid b prefixes Differential Revision: https://phab.mercurial-scm.org/D5059

context: open files in bytes mode I'm stunned this open() call has survived this long without the b in the mode - it seems like it should have been a source of bugs somewhere... Differential Revision: https://phab.mercurial-scm.org/D5080

tests: fix up test-hghave for recent run-tests change to use more CPUs Differential Revision: https://phab.mercurial-scm.org/D5085

py3: fix test-parse-date.t Differential Revision: https://phab.mercurial-scm.org/D5084

obsolete: don't translate internal error message AFAIK, it's caught only by "hg debugobsolete", so it's pretty much a programming error.

py3: get around unicode docstrings in test-encoding-textwrap.t and test-help.t On Python 3, docstrings are converted back to utf-8 bytes, which practically disables the "if type(message) is pycompat.unicode" hack in gettext(). Let's add one more workaround for the Py3 path.

crecord: make enter move cursor down to the next item of the same type Let's replace experimental.spacemovesdown with a separate key: Enter, since it wasn't used for anything in crecord. Not sure if '\n' works on Windows though. nextsametype() strictly only moves to items of the same type as the current item. This, for example, allows to go over individual lines in a diff and skip hunk and file headers (which would toggle multiple lines).

py3: fix infinitepush extension tests Differential Revision: https://phab.mercurial-scm.org/D5078

py3: build help of compression engines in bytes Removes "b''" from help.