setdiscovery: pass heads into _updatesample() In preparation for eliminating the use of dagutil. Since _takefullsample() operates on the inverted DAG, it is easier to have the caller pass in the relevant set instead of teaching _updatesample() about when to invert the DAG. We keep the logic identical for now: future commits will remove dagutil. Differential Revision: https://phab.mercurial-scm.org/D4318

setdiscovery: use a revset for finding DAG heads in a subset The march towards moving away from dagutil continues. Like other patches moving us away from dagutil, there is the potential for regressions to occur because revlogdag's headsetofconnecteds() uses revlog.index, which doesn't take filtering into account. The revset layer does. But no tests fail, so we appear to be in the clear. Differential Revision: https://phab.mercurial-scm.org/D4317

setdiscovery: reflect use of revs instead of nodes This code all operates on revision numbers. Update variable names and comments accordingly. Differential Revision: https://phab.mercurial-scm.org/D4316

dagutil: remove descendantset() and ancestorset() descendantset() is unused after the previous commit. And ancestorset() was only used by descendantset(), so it can be removed as well. .. api:: descendantset() and ancestorset() removed from dagutil Use a revset instead when operating on the changelog. Or use various functionality in the ancestor or dagop modules. Differential Revision: https://phab.mercurial-scm.org/D4315

setdiscovery: use a revset instead of dagutil.descendantset() This is the only use of descendantset() in the repo. Strictly speaking, the revset behaves slightly differently than dagutil. The reason is that dagutil is using revlog.index for DAG traversal and this data structure isn't aware of visibility / filtering. So it can operate on revisions it shouldn't operate on. But our test coverage of this code is pretty comprehensive and this change causes no tests to fail. So I think we are good. Also, the revset parser failed to parse `%ld:: - %ld::`, hence the use of descendants(). I'm not sure if that is a feature or a bug. Differential Revision: https://phab.mercurial-scm.org/D4314

setdiscovery: don't use dagutil to compute heads This is a relatively trivial operation to perform. We don't need to use dagutil. This brings us one step closer to eliminating dagutil. We still need to populate the heads on revlogdag because later functionality relies on it. Differential Revision: https://phab.mercurial-scm.org/D4310

dagutil: remove nodeset() AFAICT this was unused since the code was introduced by cb98fed52495 in 2011! Differential Revision: https://phab.mercurial-scm.org/D4309

debugcommands: use a revset instead of dagutil All this code was doing was finding the nodes that are heads from the ancestors of an input set. This can easily be expressed with a revset without having to go through dagutil. This was the last use of ancestorset() outside of dagutil itself. Differential Revision: https://phab.mercurial-scm.org/D4308

dagutil: remove internalize() and internalizeall() The previous commit removed the last consumer. .. api:: removed internalize() and internalizeall() from dagutil Use .rev(node) on storage objects to convert nodes to revisions. Differential Revision: https://phab.mercurial-scm.org/D4307

setdiscovery: don't use dagutil for node -> rev conversion The node -> rev conversion is possible using standard storage APIs and doesn't need to involve the dagutil module. Differential Revision: https://phab.mercurial-scm.org/D4306

dagutil: remove externalize() and externalizeall() They are unused after the previous commit. .. api:: externalize() and externalizeall() removed from dagutil Use .node() on a storage primitive to perform revision to node conversions. Differential Revision: https://phab.mercurial-scm.org/D4305

setdiscovery: don't use dagutil for rev -> node conversions We don't need to use dagutil to perform a simple rev -> node conversion. I haven't measured, but the new code is likely faster, as we avoid extra function calls and avoid some attribute lookups. Differential Revision: https://phab.mercurial-scm.org/D4304

exchange: don't use dagutil We were only using it for simple node -> rev and parent revision lookups. These are exposed via the storage interface and we don't need to go through dagutil. Differential Revision: https://phab.mercurial-scm.org/D4303

revlog: only consider the span of the delta section Since the number of snapshots is limited we can exclude them from the logic checking size and number of reads. Limiting the span computation to the delta section will allow for further optimization.

revlog: ensure intermediate snapshot have decreasing size If the intermediate snapshot is bigger than the previous one, there is likely a better snapshot to be made at a different level.

revlog: bound number of snapshots in a chain To limit the number of snapshot chained, we enforce them to be smaller and smaller. This guarantee the number of snapshot in a chain will be bounded to a small number.

revlog: compute snapshot depth on delta info We need the information to be available when choosing delta.

debugrevlog: display snapshot details per depth This help in understanding the final structure of build manifest. All data about snapshot (full and intermediate) are gathered into a sub-list for clarity. Since we do not produce such snapshots yet, the only thing changing in test output is the way the information is presented.

revlog: add a method to retrieve snapshot depth Some snapshot property (eg: maximum size) will depend on their depth.

debugrevlog: include information about intermediate snapshots As we are about to create intermediate snapshots, we need to have a way to debug them. We start by adding very simple debug output and more detailed output will comes in next changesets.

revlog: also detect intermediate snapshots Also detect intermediate-snapshot done against another previous snapshot. Doing an intermediate snapshot instead of a full one can reduce the number of full snapshots we need. They are especially useful for content with a lot of churn on the same line (eg: the manifest) where having a delta over multiple revisions can end up being significantly smaller than the sum of these revision deltas. A revlog built using intermediate snapshots can be a bit smaller and reuse snapshot much more efficiently. This last property is useful combined with constraints on chain length. Using intermediate snapshot can produce repository with delta chain ten times shorter without impact on the storage size. Shorter chain lengths are faster to restore, greatly improving read performance. This changesets (and the following ones) focus on getting the core principle of intermediate snapshots into Mercurial core. Later changeset will introduce the strategy to create them.

revlog: add a method to tells whether rev is stored as a snapshot For now we only have one type of snapshot: full snapshot versus nullrev. However we are looking into adding intermediate snapshot where a large diff against another snapshot is performed instead of storing a full new text. The conditional is a bit strange and is done in order to help readability of a some later changesets.

debugrevlog: fix for non-manifest object The `filelog` object is no longer an actual revlog. Instead, the actual revlog is stored in the `_revlog` attribute.

merge with stable

nodes: expand/comment the magic nodes so they are more easily searchable We just encountered `000000000000modified`, and it was quite annoying to search for these, even though I knew they existed. For those that don't know that they exist, this is essentially impossible to search for :) (Technically we encountered it in its hex form, 3030303030303030303030306d6f646966696564, so I'm adding comments with those forms in case that's helpful to people in the future). Differential Revision: https://phab.mercurial-scm.org/D4331

revlog: obtain the first node at the lowest layer while building pure nodemap Just for clarity. This doesn't matter in practice since changelog.nodemap is accessed *before* filtered revisions get ready.

revlog: fix pure nodemap to not access missing index entry This bug was revealed by a3dacabd476b and a1f934573c0b.

changelog: remove copy of revlog.nodemap() It's been there since 2012, "clfilter: introduce `filteredrevs` attribute on changelog." I don't think we can apply changelog filtering to nodemap at this level, so this patch removes the nodemap stub completely.

branchmap: explicitly convert file into iterator Follows up 2a4bfbb52111. This is required for httprangereader, which is not an iterable itself.

branchmap: close cache file properly Follows up 2a4bfbb52111.

branchmap: strip '\n' read from cache file as before Follows up 2a4bfbb52111.

rebase: do not pass in user option to rollback in-memory merge conflict Nothing passed before e9e742bd0501.

rebase: add test for in-memory merge conflicts

rebase: call _dorebase() properly This fixes a regression from e9e742bd0501 where we failed to pass all necessary arguments to _dorebase(). Differential Revision: https://phab.mercurial-scm.org/D4302

context: make sure file is not deleted while checking path conflicts If a file is deleted and a directory of same name is created in the same commit, IMM thinks of that as a file conflict, however the file is deleted and hence the directory can be created. The test change demonstrate the fix. Differential Revision: https://phab.mercurial-scm.org/D4300

tests: demonstrate that IMM needs to be smarter with path conflicts When we try to rebase a commit which deletes an existing file and make a directory of the same name, rebase with IMM aborts. It should work fine just like the without IMM case. Differential Revision: https://phab.mercurial-scm.org/D4299

tests: don't create new repo inside existing repo in test-rebase-inmemory.t Differential Revision: https://phab.mercurial-scm.org/D4298

tests: remove test-py3-commands.t This was a smoke test for early in the Python 3 porting effort, before anything actually worked. Now that we've got over half the testsuite passing, this test has outlived its utility. Differential Revision: https://phab.mercurial-scm.org/D4288

tests: update test-check-py3-compat.t output in the py3exe branch This hasn't been maintained in a while, it looks like. Differential Revision: https://phab.mercurial-scm.org/D4289

overlayworkingctx: fix exception in metadata-only inmemory merges (issue5960) If there was a metadata-only mutation, such as +x or -x on a file, we would create a cache entry with None for data, and this would cause problems later on when some code tried to run fctx.data() or similar, and was expecting a string. My original fix for this involved passing data=self._wrappedctx[path].data() in setflags(), but this version seems slightly better - this way, if we ever call write() and then call setflags(), we don't destroy the data that we wrote that's in the cache. I haven't verified that other fields aren't destroyed, such as date or flags :) Differential Revision: https://phab.mercurial-scm.org/D4287

filemerge: show actual capabilities of internal merge tools This information is useful to know which internal merge tools can be applied safely on binary files and/or symlinks.

filemerge: add config knob to check capabilities of internal merge tools For historical reason, Mercurial assumes capabilities of internal merge tools as below while examining rules to decide merge tool, regardless of actual capabilities of them. =============== ====== ======== specified via binary symlinks =============== ====== ======== --tool o o HGMERGE o o merge-patterns o (*) x (*) ui.merge x (*) x (*) =============== ====== ======== This causes: - unintentional internal merge tool is chosen for binary files via merge-patterns section of configuration file - explicit configuration of internal merge tool for symlinks is ignored unintentionally But on the other hand, simple "check capability strictly" might break backward compatibility (e.g. existing merge automations), because it changes the result of merge tool selection. Therefore, this patch adds config knob "merge.strict-capability-check" to control whether capabilities of internal merge tools should be checked strictly or not. If this configuration is true, capabilities of internal merge tools are checked strictly in (*) cases above.

filemerge: show warning if chosen tool has no binary files capability While matching patterns in "merge-patterns" configuration, Mercurial silently assumes that all merge tools have binary files capability. This implementation comes from 5af5f0f9d724 (or Mercurial 1.0). At failure of merging binary files with incorrect internal merge tool, there is no hint about this silent ignorance of binary files capability. This patch shows warning message, if chosen internal merge tool has no binary files capability. This will help users to investigate why a binary file isn't merged as expected.

filemerge: add the function to examine a capability of a internal tool For "symlink" and "binary" capabilities, _toolbool() can not examine these of internal merge tools strictly, because it examines only configurations in "merge-tools" section. Users can configure them explicitly as below for example, but this is not ordinary usage and not convenient: [merge-tools] :other.symlink = true :other.binary = true This patch adds hascapability() internal function, which can examine actual capabilities of a internal merge tool strictly. At this patch, hascapability() is still used with "strict=False". Subsequent patches use it with "strict=True".

filemerge: set actual capabilities of internal merge tools This information is used to detect actual capabilities of internal merge tools by subsequent patches. For convenience, this patch assumes that merge tools typed as "nomerge" have both binary files and symlinks capabilities.

help: describe more detail about capabilities while deciding merge tool "hg help merge-tools" describes as below: (internal merge tools) will by default not handle symlinks or binary files. But in some cases, Mercurial assumes that internal merge tools have one or both of these capabilities. "hg help merge-tools" also describes as below, for matching patterns in merge-patterns configuration section. But this is not sufficient. Here, binary capabilities of the merge tool are not considered. This patch describes more detail about capabilities while deciding merge tool.

tests: un-glob patchbomb test lines that were only globbing a hostname Differential Revision: https://phab.mercurial-scm.org/D4296

tests: force a stable hostname in patchbomb tests No visible output changes in this commit because everything is globbed away, but on Python 3 the stable hostname will mean that the headers don't trigger RFC2822 multi-line mode because they'll always be consistently short. Differential Revision: https://phab.mercurial-scm.org/D4295

patchbomb: allow using HGHOSTNAME to force a hostname I'll update run-tests.py to set this globally to stabilize some tests. The variable name is intentionally generic because I suspect we should generalize this to other tests. Differential Revision: https://phab.mercurial-scm.org/D4294

patchbomb: extract function for generating message-id Differential Revision: https://phab.mercurial-scm.org/D4293

rebase: cover restorestatus() by lock to prevent it from being updated To prevent it from being updated by another process `restorestatus()` is moved under lock. Differential Revision: https://phab.mercurial-scm.org/D4282

branchmap: load branchmap as an iterable This avoids reading all the file into memory if the cache turns out to be invalid. Differential Revision: https://phab.mercurial-scm.org/D4281

perf: time loading branchmap caches Differential Revision: https://phab.mercurial-scm.org/D4280

tests: add conditional output when simplestore extensions is loaded This drops the number of failures with this extension to 3. Differential Revision: https://phab.mercurial-scm.org/D4286

tests: conditionalize extension tests for extra extensions If extra extensions are loaded (e.g. via --extra-config-opt), the tests conditionalized in this commit fail in ways that are dependent on the extensions that are loaded. So let's skip them when that scenario is present. This drops the number of failures for the simplestorerepo.py extension to 4. Differential Revision: https://phab.mercurial-scm.org/D4285

commit: try hard to reuse p1 manifest if nothing changed This is all for commit reproducibility on "hg convert". With this change, p1 manifest is reused if ctx.files() *to be committed* is empty, and if new manifest entry is identical to p1. This is important property for "hg convert" since memctx.files() built from a convert source may be either a) more narrowed thanks to a committed ctx.files() which provides more accurate status, or b) containing redundant files because of sloppy filtering on e.g. octopus merge.