merge: default into stable for release candidate

debugdiscovery: include the number of heads in all sets We already displayed information about heads of the common set that are either local or remote heads. We now also do so for heads of the common set that are both local and remote heads too. This is useful because various step in the set discovery algorithm have head specific optimizations.

recover: add a --[no-]verify flag For trivial cases, the cost of the verify run after `hg recover` is getting in the way. In addition for very large repositories, the cost is simply too high to be paid, making `hg recover` an unusable commands. We introduce a --verify flag, set by default. If is automatically associated with a --no-verify flag that one can use to skip the verify step. We might consider changing the default behavior in the future. However this is out of scope for this series.

bundle2: handle compression in _forwardchunks _forwardchunks is used to compensate for getbundle protocol deficits. Since it transparently decodes the payload, it also needs to remove the corresponding compression parameter in case the server decides to send one. This the wire protocol part of issue 5990. Differential Revision: https://phab.mercurial-scm.org/D6182

changelog: parse copy metadata if available in extras This lets read back the copy metadata we just started writing. There are still many places left to teach about getting the copy information from the changeset, but we have enough ({file_copies}, specifically) that we can add it now and have some test coverage of it. Differential Revision: https://phab.mercurial-scm.org/D6186

copies: add config option for writing copy metadata to file and/or changset This introduces a config option that lets you choose to write copy metadata to the changeset extras instead of to filelog. There's also an option to write it to both places. I imagine that may possibly be useful when transitioning an existing repo. The copy metadata is stored as two fields in extras: one for copies since p1 and one for copies since p2. I may need to add more information later in order to make copy tracing faster. Specifically, I'm thinking out recording which files were added or removed so that copies._chaincopies() doesn't have to look at the manifest for that. But that would just be an optimization and that can be added once we know if it's necessary. I have also considered saving space by using replacing the destination file path by an index into the "files" list, but that can also be changed later (but before the feature is ready to release). Differential Revision: https://phab.mercurial-scm.org/D6183

revsetbenchmark: add some simpler revset for heads and roots This revset might leverage compiled code in the future so lets start to track them.

repoview: flag `server.view` as experimental Ideally, the non-experimental version of `experimental.extra-filter-revs` will cover the use case for `server.view=immutable` well enough than having to have this dedicated configuration. Since `server.view` is not part of any release, I would prefer to have it marked as experimental to avoid having it to support it for ever.

repoview: move subsettable in a dedicated module The dictionary got moved in `branchmap` to avoid import cycle. However, we are about to needs it in repoview too. So we introduce a now module to define that that mapping.

upgrade: support upgrade to/from zstd storage (issue6088) Now that we have an official config option for a shiny format improvement, we better make it simple to migrate to/from it.

compression: introduce an official `zstd-revlog` requirement This requirement supersedes `exp-compression-zstd`. However, we keep support for the old requirement. Strictly speaking, we do not need to add a new requirement, there are no logic change making "new" repo incompatible with mercurial client using a version that support `exp-compression-zstd`. The choice to introduce a new requirement is motivated by the following: * The previous requirement was explicitly "experimental". Using it by default could confuse users. * adding support for a hypothetical third compression engine will requires new code, and will comes with its own requirement tool. * We won't use it as the default for a while since I do not think we support zstd on all platform. I can imagine we'll gain another (unrelated but on my default) requirement by the time we turn this zstd by default.

compression: only declare revlog support for available engine Even if we know that an engine support revlog compression, the Mercurial process still won't support it if the compression engine is not available.

branchcache: lazily validate nodes in iteritems() This saves ~0.30 sec on creating a new branch on our internal repo. Differential Revision: https://phab.mercurial-scm.org/D6236

branchcache: only iterate over branches which needs to be verified Otherwise we loop over all the branches and call _verifybranch() even if not required. Differential Revision: https://phab.mercurial-scm.org/D6240

branchcache: fix the docstring of _verifybranch() Initially the function was designed to support verifying all branches but later I decided to have a separate function. I forget to remove the doc related to that. Thanks to Yuya for spotting this in review. Differential Revision: https://phab.mercurial-scm.org/D6239

branchcache: don't verify while creating a copy The copy will not be mark as verified, so there is no need to verify nodes. Thanks to Yuya who spotted this while reviewing. Differential Revision: https://phab.mercurial-scm.org/D6238

pull: improved message issued in case of failed update When running `hg pull --update`, the update step may fail. Nothing in the error message help to understand the abort is related to the secondary step (update) instead of the primary step (pull). We now add some information to the error message to clarify it comes from the update part. It is useful in various situation (uncommitted changes blocking the update, update to hidden destination, etc...) The pull output is updated from: $ hg pull ../repo-Bob --rev 956063ac4557 --update pulling from ../repo-Bob searching for changes adding changesets adding manifests adding file changes added 2 changesets with 0 changes to 2 files (+1 heads) (2 other changesets obsolete on arrival) abort: filtered revision '6'! to: $ hg pull ../repo-Bob --rev 956063ac4557 --update pulling from ../repo-Bob searching for changes adding changesets adding manifests adding file changes added 2 changesets with 0 changes to 2 files (+1 heads) (2 other changesets obsolete on arrival) abort: cannot update to target: filtered revision '6'! (I am not sure why the actual error, "filtered revision '6'", is not using the more modern format mentioning the obsolescence fate of '6')

pull: deal with locally filtered changeset passed into --rev Nowadays, it is possible to explicitly pull a remote revision that end up being hidden locally (eg: obsoleted locally). However before this patch, some internal processing where crashing trying to resolve a filtered revision. Without this patches, the pull output result a confusing output: $ hg pull ../repo-Bob --rev 956063ac4557 pulling from ../repo-Bob searching for changes adding changesets adding manifests adding file changes added 2 changesets with 0 changes to 2 files (+1 heads) (2 other changesets obsolete on arrival) abort: 00changelog.i@956063ac4557828781733b2d5677a351ce856f59: filtered node!

absorb: aborting if another operation is in progress This increases safety of using absorb by both aborting when another operation is in progress (since the absorption could confuse any other command a lot) and holding the locks throughout the reading of the working directory (for which changes to absorb) and the reading of the repo (for which changes to absorb into). Differential Revision: https://phab.mercurial-scm.org/D6237

repoview: keep the branchmap cache for the `served.hidden` view warm For the same reason we want to keep the cache for the `served` view up to date, we want to also keep the `served.hidden` view up to date. If some processes with a readonly access to the repo needs to access it, we better have the cache warm to avoid computing the same data over and over and over. In most case (no secret changesets), the "served" and "served.hidden" set will be identical and no cache will actually have to be updated.

repoview: introduce a filter for serving hidden changesets There are multiple usecase for being able to explicitly view or pull obsolete from a server. We need to be able to do so without exposing the secret changesets. We introduces a dedicated repository "view" to do so. Way to expose this "view" to the user will come later. To keep a behavior consistent with expected client/server behavior, the general idea is for the obsolete access to be explicitly requested by the code generating the request. In addition, the will be server side configuration to restrict the access to this feature.

repoview: fix conditional around unserved changesets The conditional could lead to wrong computation since we have more unserved changesets than just the "secret" phase. In addition, now that we have efficient caching of phased changesets, we don't need the conditional anymore.

branch: return early if we find an open named branch apart from default The current code builds a list of all the open named branches except default and then bool that. This is mostly fine until you get a repo which has thousands of named branches. Differential Revision: https://phab.mercurial-scm.org/D6211

branchcache: don't verify closed nodes in _branchtip() We only do membership testing there. Differential Revision: https://phab.mercurial-scm.org/D6210

branchcache: don't verify closed nodes in iteropen() We expect that the nodes passed to iteropen() will be verified. We are only testing for membership in closed nodes set, so we don't need to verify the whole closed nodes set. This will speed up calculating branchheads() when there are lot of closed nodes related to other branches. Differential Revision: https://phab.mercurial-scm.org/D6209

branchcache: lazily validate nodes from the branchmap On my personal hg-repository with 365 entries in .hg/cache/branch2, following are the numbers for perfbranchmapload. Before this patch: ! wall 0.000866 comb 0.000000 user 0.000000 sys 0.000000 (best of 2680) ! wall 0.001525 comb 0.000000 user 0.000000 sys 0.000000 (max of 2680) ! wall 0.001107 comb 0.001097 user 0.001086 sys 0.000011 (avg of 2680) ! wall 0.001104 comb 0.000000 user 0.000000 sys 0.000000 (median of 2680) With this patch: ! wall 0.000530 comb 0.000000 user 0.000000 sys 0.000000 (best of 4240) ! wall 0.001078 comb 0.000000 user 0.000000 sys 0.000000 (max of 4240) ! wall 0.000696 comb 0.000693 user 0.000677 sys 0.000017 (avg of 4240) ! wall 0.000690 comb 0.000000 user 0.000000 sys 0.000000 (median of 4240) On our internal repository with ~20k entries in branchcache, I see improvement from 0.125 sec to 0.066 sec which is 47% speed up. The above are the numbers of perfbranchmapload which shows how much time we saved by not validating the nodes. But we need to validate some nodes. Following are timings of some mercurial operations which have speed up because of this lazy validation of nodes: No-op `hg update` on our internal repository (Avg on 4 runs): Before: 0.540 secs After: 0.430 secs Setting a branch name which already exists without --force (Avg of 4 runs): Before: 0.510 secs After: 0.250 secs I ran the ASV performance suite and was unable to see any improvements except there was improvement of perfdirstatewrite() on netbeans which I think was not related. I looked into the commit code, the command which I am trying to speedup, it looks like it uses revbranchcache to update the branchcache. Differential Revision: https://phab.mercurial-scm.org/D6208

branchcache: add functions to validate changelog nodes This patch adds functions to validate closed nodes, validate nodes for a certain branch and for all the branches. These functions will be used in upcoming patches. Differential Revision: https://phab.mercurial-scm.org/D6207

readline: provide styled prompt to readline (issue6070) Differential Revision: https://phab.mercurial-scm.org/D6168

copies: move comment about implementation of mergecopies() to end When you start reading about mergecopies(), you want to know what it is, not that there are different implementations depending on config, so this patch moves that comment to the end. By the way, we don't seem to define what "copytracing" is. I'm just leaving it that way because I don't know what it is myself. It seems to be referred to only on mergecopies() (and not in pathcopies(), for example), so maybe "copytracing" is supposed to be exactly what mergecopies() does? Differential Revision: https://phab.mercurial-scm.org/D6235

remotefilelog: return expected type from copies overrides copies._computeforwardmissing() and copies._computenonoverlap() return sets, so the overrides should also do that. Differential Revision: https://phab.mercurial-scm.org/D6234

changelog: extract a _string_unescape() to mirror _string_escape() We use our own _string_escape() to encode the "extras" field. Then we use codecs.escape_decode() to escape it. But there's also a little workaround for dealing with escaped text that looks like octal numbers since the fix for https://bz.mercurial-scm.org/show_bug.cgi?id=3156. This patch extracts the call to codecs.escape_decode() along with the fix for octal numbers and puts it in a _string_unescape(). It also updates the test to check for the octal-number case from the aforementioned bug. As you may have suspected, I want to be able to reuse this new function later. Differential Revision: https://phab.mercurial-scm.org/D6184

copies: extract function for deciding whether to use changeset-centric algos We'll eventually have a "experimental.copies.read-from=changeset-only" option too and I don't want to spread the logic for determining if we should use changeset-centric of filelog-centric algorithms. Differential Revision: https://phab.mercurial-scm.org/D6163

getrenamedfn: get copy data from context object if configured The function returned from getrenamedfn() calls filelog.renamed(). That won't work when storing copy metadata in the changeset. I've just switched to a simple implementation here. We may or may not need to optimize it later, possibly by optimizing the callers. No more tests fail with "--extra-config-opt experimental.copies.read-from=compatibility)" than they did before this patch. Differential Revision: https://phab.mercurial-scm.org/D6162

branchmap: implement __contains__() We have good occurences of `if branch in branchmap()` in our code. If __contains__() is not implemented then it will use __iter__() to find whether the element exists or not which is not good. I am bit confused that whether I should move existing callers to hasbranch() or this patch is a good way. Differential Revision: https://phab.mercurial-scm.org/D6206

branchmap: prevent using __getitem__() in branchheads() branchheads() can directly use self._entries instead. Differential Revision: https://phab.mercurial-scm.org/D6205

branchmap: dynamically resolve type of branchcache class This is required to support subclassing. Thanks to Yuya for suggesting this in D6151. Differential Revision: https://phab.mercurial-scm.org/D6204

copies: print list of divergent renames in sorted order The current order is not obvious to the user (and I may change it in a later patch). Lexicographical order seems like the obvious choice here. Differential Revision: https://phab.mercurial-scm.org/D6227

import: suppress accept: header irccloud and others may perform content negotation and reject (406) if the accept header does not match the mime type of the object

rebase: fix bug that prevented dry-run rebases from printing failures As far as I can tell it should be fine to unconditionally skip _prepareabortorcontinue if we're in the process of raising an Abort here. Differential Revision: https://phab.mercurial-scm.org/D6226

rebase: demonstrate bug in dry-run mode which causes cycles to not be reported Differential Revision: https://phab.mercurial-scm.org/D6225

test: minor cleanup to test-server-view.t While looking into adding error output in this test, I did some cleanup.

repoview: improve documentation for `repo.filtered` method I am sitting next to Joerg Sonnenberger and we are discussion his experience with repoview. This first effect of this discussion is this documentation clarification.

revset: short docstring for checkstatus This is where all the action happens for the status-related revsets, and a little documentation doesn't hurt.

discovery: stop direct use of attribute of partialdiscovery Instead of accessing `undecided` directly for ui display purposes, we introduce a `stats()` method that could be extended in the future with more interesting information. This is in preparation for a forthcoming Rust version of this object. Indeed, attributes and furthermore properties are a bit complicated for classes in native code. We could go further and rename `undecided` to mark it private, but `_undecided` is already taken as support for `_undecided` lazyness.

overlayworkingctx: remove misleading trailing slash from directory pattern The paths passed into the matcher are normalized (this applies to include patterns and regular patterns, and to both glob kind and path kind), so the regex for input "foo/" ended up being "foo(?:/|$)". Once we have a (recursive) pattern kind only for directories, we could switch to that here and remove the "mfiles[0] == path" check. Until then, let's at least make it not misleading. Differential Revision: https://phab.mercurial-scm.org/D6224

overlayworkingctx: fix file/dir audit to be repo-relative Before this patch, test-rebase-inmemory.t would stop erroring out about the conflict if you added a "cd a" before line 252. That was because a glob matcher (which are relative) was unintentionally used. That happened because the matcher was given "include" patterns (not regular patterns), and "include" patterns are always glob by default (i.e. unless you write them including the kind prefix). IOW, the "default='path'" argument passed to ctx.match() was ignored. Differential Revision: https://phab.mercurial-scm.org/D6223

messages: replace some instances of "folder" by "directory" I'm pretty sure this is our preferred term. Differential Revision: https://phab.mercurial-scm.org/D6222

match: fix re2 compability broken in 2e2699af5649 When using re2, we call test_match() instead of match() on the compiled regex object. While match() returns a matcher object or None, test_match() returns True or False. So since 2e2699af5649 running test on a machine with a re2 install fails in many places. Instead we make the code a bit more general and everything goes back to normal.