cleanupnodes: pass multiple predecessors to `createmarkers` directly

obsolete: allow multiple predecessors in createmarkers The logic for this change is similar to the change to `cleanupnodes` that we did earlier. Now that the rebase code is trying to record a fold, we need to actually record it in the markers. The first step is to have the markers creation API able to receive such fold data. To keep things sane, we restrict fold to on successors.

obsolete: preindent code in createmarkers This will help to make the next change clearer.

obsolete: explicitly pass relation items to effectflag computation To track folds, we are about to change the content of `rel`. To simplify this change, we update the `geteffectflag` function beforehand.

rebase: explicitly track collapses as fold A collapse is a large fold. It is now explicitly tracked (at the cleanupnodes API level).

rebase: use tuple as `replacement` keys Now that `cleanupnodes` support tuples as key, we update the rebase code to use them. No changes in the replacement tracked are introduced yet.

rebase: expand a long "one-liner" When a one-liner gets 3 lines longs, it lose its expressivity benefits. We expand it into a simple for loop. This makes future changes of the code in that area clearer.

cleanupnodes: drop special casing around prune markers (API) The `cleanupnodes` has logic to skip the creation of "prune" markers if the changeset is already obsolete. This feels strange and gets in the way of code changes to tracks folds. Now that callers no longer request such prune, we can drop this logic. In many cases, pruning through cleanupnodes should be replaced by internal phase usage.

rebase: don't try to prune obsolete changeset already in the destination With similar motivations to the previous changesets, we stop marking changeset from pruning when it is not the command intention. In this case, we still need to distinguish between the strip and the obsolete case.

histedit: don't cleanup nodes already disposed of If something else took care of these temporary nodes, we don't need to do anything about it. This less liberal usage of pruning through cleanup nodes will help us further cleanup on the road to explicitly tracks folds.

py3: avoid b'' output in test-hgweb-non-interactive.t I couldn't figure out how to get the list to print without b'' with pycompat.write(), without converted each element to str.

py3: convert print to a function call in a few tests I *think* this is the last of them.

py3: use util.forcebytestr() to convert push lock error to bytes

py3: byteify windows.shelltocmdexe() This makes test-doctest.py happy on Windows.

run-tests: flush output stream before prompting to accept changes With py3 on Windows, the prompt to accept changes in an --interactive run wasn't showing up until after the choice was read from stdin.

py3: use util.forcebytestr() to convert error messages to bytes This makes the python 3 buildbot green again. Differential Revision: https://phab.mercurial-scm.org/D4811

context: stop catching TypeError when converting hex nodeid to binary It has been a programming error to pass a 40-character string that is not a hex nodeid since 8b86acc7aa64 (context: drop support for looking up context by ambiguous changeid (API), 2018-04-28), so we can just let it raise a TypeError. Differential Revision: https://phab.mercurial-scm.org/D4807

context: stop catching and re-raising FilteredRepoLookupError FilteredRepoLookupError is only raised by changectx's constructor and the higher-level scmutil.revsymbol(), so there's no need to catch it in changectx's constructor. Differential Revision: https://phab.mercurial-scm.org/D4806

util: use a context manager in readlock()

py3: convert os.readlink() path to native strings on Windows Windows insisted that it needs to be str. I skipped the stuff in the posix module, and left `tests/f` and `run-tests.py` alone for now.

py3: suppress the output from .write() calls in the remaining tests

py3: conditionalize json float precision difference in test-debugcommands.t

py3: byteify extension in test-debugcommands.t

pullreport: rev duplicated and extinct into account If we already have some obsolete and hidden nodes locally and the server send them again to you, it seems useful to point it out instead of being silent about it.

pullreport: issue a message about "extinct" pulled changesets Changeset pulled from a remote repository while already obsolete locally can end up hidden after the pull. Hiding obsolete changesets is a good behavior but silently "skipping" some of the pulled content can get confusing. We now detect this situation and emit a message about it. The message is simple and the wording could be improved, however, we focus on the detection here. Evolution is still an experimental feature, so the output is open to changes. In particular, we could point out at the latest successors of the obsolete changesets, however, it can get tricky is there are many of them. So we delay these improvements to another adventure. Another easy improvement would be to merge this message with the previous line about the new nodes and their phases. This is a good example of cases where we can only transmit a limited amount of data to users by default. We need some sort of "transaction journal" we could point the user to.

pullreport: skip or rework some early return We are about to add more logic in this report. Before that, we need it to not quit so early.

pullreport: skip filtered revs instead of obsolete ones Obsolescence is closely related to visibility but still a distinct concept. We can receive changesets that are obsolete but visible (eg: when pulling orphans). Such changeset should be reported too. In addition, the filtering level can be anything, we should respect it.

pullreport: add a test to show misreporting of visible changeset The current code ignores all obsolete changesets including the visible one. We add a test showing this behavior before fixing the behavior.

repo: don't look up context for tip node if it's not needed We were doing repo['tip'].node() or similar in a few places where repo.changelog.tip() would be enough. Differential Revision: https://phab.mercurial-scm.org/D4781

repo: look up nullrev context by revnum, not symbolic name I think lookup of the 'null' symbol should be done via scmutil.revsymbol() and repo['null'] is only supported for historical reasons. However, repo[nullrev] is fine, so we can switch to that instead of switching to scmutil.revsymbol('null'). Differential Revision: https://phab.mercurial-scm.org/D4780

bundle: consistently put revnums in "base" collection The "base" collection contains revnums, except that it can also contain the "null" symbol. That's a little weird. Let's be consistent. Differential Revision: https://phab.mercurial-scm.org/D4779

py3: ensure printing to stdout uses str in test-hgweb-no-request-uri.t

scmutil: accept multiple predecessors in 'replacements' (API) This changeset makes 'cleanupnodes' accepts multiple predecessors as `replacements` keys. The same as it accepts multiple successors as `replacements` values. To avoid breaking all callers, the old and new ways are currently valid at the same time. We'll deprecate and drop the old way later. This change is the first step toward a better tracking of "fold" event in the evolution history. While working on the "rewind" command (in the evolve extension), we realized that first class tracking of folds are necessary. We already have good tracking of splits. When walking the evolution history from predecessors to successors, that makes for a clear distinction between having multiple successors because of the actual splitting of a changeset or content-divergences. The "rewind" command allows restoring older evolution of a stack of changesets. One of its mode walks the evolution history to automatically find appropriate predecessors. This means walking from successors to predecessors. In this case, we need to be able to make the same distinction between an actual fold and other cases. So we will have to track folds explicitly. This changesets only focus on making it possible to express fold at the `cleanupnodes` API level. The actual tracking will be implemented later.

scmutil: expand long "one-liner" When a one-liner gets 3 lines longs, it loses its expressivity benefits. We expand it into a simple for loop. This makes future changes of the code in that area clearer.

shelve: find shelvedctx from bundle even if they are already in the repo We use the new "duplicates" node tracking to find the tip of the bundle even if it already exists in the repository. Such logic is not supposed to be needed in theory. If the shelve was made using internal-phase, we already know its node. Otherwise, the bundle content should have been stripped. However, handling it makes the shelve code more robust and provide a good example of "revduplicates" usage.

shelve: return the shelved node as part of bundle application It make sense to have the function in charge of unbundling the shelved revision also return the node of that revision (when the data is in the bundle). This will help us to handle unnatural state where the unshelved change already exists in the repository.

changelog: keep track of duplicated node in the transaction adding them The transaction is already tracking the new nodes. We now tracks the "duplicates" in the same location.

revlog: add a callback "tracking" duplicate node addition If a changegroup contains node already added to the repository, they will be skipped. Skipping them is the right behavior (we don't need to store things twice), but it can hide some information to the code doing the unbundle (eg: shelve looking for the tip of the bundle). The first step to improve this situation is to add a low level callback. We do not need this tracking on all revlog, so actual tracking will be added in the next changeset.

logtoprocess: define $HG for children processes So they can compute the hg version for instance. Differential Revision: https://phab.mercurial-scm.org/D4768

py3: mask out None type when printing in `debuglocks` Apparently, %b doesn't allow None.

py3: ensure standard exceptions use `str` type strings in windows.py See also edaa40dc5fe5.

py3: replace a StandardError reference This doesn't exist on py3, and the standard way of handling this seems to be to catch both exceptions.

storageutil: extract revision number iteration This code is a bit quirky (and possibly buggy). It will likely be used by multiple storage backends. Let's extract it so it is reusable. Differential Revision: https://phab.mercurial-scm.org/D4757

storageutil: new function for extracting metadata-less content from text Other storage backends will want to do this. I'm not concerned about Python function call overhead because I expect self.revision() to dwarf the function call overhead time, since self.revision() requires multiple function calls and may involve decompression in the common case. Differential Revision: https://phab.mercurial-scm.org/D4756

storageutil: move _censoredtext() from revlog This seems like generic functionality we'll want to use from non-revlog storage backends. Differential Revision: https://phab.mercurial-scm.org/D4755

storageutil: move metadata parsing and packing from revlog (API) Parsing and writing of revision text metadata is likely identical across storage backends. Let's move the code out of revlog so we don't need to import the revlog module in order to use it. Differential Revision: https://phab.mercurial-scm.org/D4754

storageutil: new module for storage primitives (API) There will exist common code between storage backends. It would be nice to have a central place to put that code. This commit attempts to create that place by creating the "storageutil" module. The first thing we move is revlog.hash(), which is the function for computing the SHA-1 hash of revision fulltext and parents. Differential Revision: https://phab.mercurial-scm.org/D4753

filelog: stop proxying deltaparent() (API) deltaparent() obtains the revision number of the base revision a delta in storage is stored against. It is highly revlog-centric and may not apply to other storage backends. As a result, it doesn't belong on the generic file storage interface. This method/proxy is no longer used in core. The last consumer was probably changegroup code and went away with the transition to emitrevisions(). Differential Revision: https://phab.mercurial-scm.org/D4751

filelog: stop proxying rawsize() (API) This method is no longer used by external consumers. The API is quite low-level and is effectively len(revision(raw=True)). I don't see a compelling reason to keep it around. Let's drop the API and make the file storage interface simpler. Differential Revision: https://phab.mercurial-scm.org/D4750

filelog: stop proxying "opener" (API) The last consumer of it in upgrade code was removed as part of the previous commit. This attribute is revlog specific (because it assumes the existence of a vfs for performing I/O on tracked file data) and therefore isn't appropriate for a generic storage interface. So nuke it. Differential Revision: https://phab.mercurial-scm.org/D4749

filelog: stop proxying flags() (API) Per-revision storage flags are kinda a revlog-centric API. (Except for the fact that changegroup uses the same integer flags as revlog does and there's minimal verification that the server's flags map to the client's storage flags - but that's another problem.) The last user of flags() was in verify.py and that code was just moved into revlog.py and is accessed behind the verifyintegrity() file storage API. Since there are no more consumers, let's drop the proxy and remove the method from the file storage interface. This commit only drops the dedicated API for reading a single revision's storage flags: we still support reading and writing flags through the bulk data retrieval and add revision APIs. And since changegroups encode revlog integer flags over the wire, we'll always need to support flags at some level. The removal of individual storage flags may be too premature. But since flags() is now unused, I'd like to see how far we can get without that dedicated API - especially since it uses revision numbers instead of nodes. Differential Revision: https://phab.mercurial-scm.org/D4746

revlog: move revision verification out of verify File revision verification is performing low-level checks of file storage, namely that flags are appropriate and revision data can be resolved. Since these checks are somewhat revlog-specific and may not be appropriate for alternate storage backends, this commit moves those checks from verify.py to revlog.py. Because we're now emitting warnings/errors that apply to specific revisions, we taught the iverifyproblem interface to expose the problematic node and to report this node in verify output. This was necessary to prevent unwanted test changes. After this change, revlog.verifyintegrity() and file verify code in verify.py both iterate over revisions and resolve their fulltext. But they do so in separate loops. (verify.py needs to resolve fulltexts as part of calling renamed() - at least when using revlogs.) This should add overhead. But on the mozilla-unified repo: $ hg verify before: time: real 700.640 secs (user 585.520+0.000 sys 23.480+0.000) after: time: real 682.380 secs (user 570.370+0.000 sys 22.240+0.000) I'm not sure what's going on. Maybe avoiding the filelog attribute proxies shaved off enough time to offset the losses? Maybe fulltext resolution has less overhead than I thought? I've left a comment indicating the potential for optimization. But because it doesn't produce a performance regression on a large repository, I'm not going to worry about it. Differential Revision: https://phab.mercurial-scm.org/D4745

tests: de-flake test-narrow-debugrebuilddirstate.t If the dirstate gets written much later (usually 1-2 s, depending on FS) than the working copy file (there's only one), then the `hg debugdirstate` command will include a timestamp. There's nothing wrong with that, so we should just allow it. Differential Revision: https://phab.mercurial-scm.org/D4758

upgrade: use storageinfo() for obtaining storage metadata Let's switch to our new API for obtaining information about storage. This eliminates the last consumer of rawsize() and the opener proxy from the file storage interface! Differential Revision: https://phab.mercurial-scm.org/D4748

revlog: add method for obtaining storage info (API) We currently have a handful of methods on the file and manifest storage interfaces for obtaining metadata about storage. e.g. files() is used to obtain the files backing storage. rawsize() is to quickly compute the size of tracked revisions without resolving their fulltext. Code in upgrade and stream clone make heavy use of these methods. The existing APIs are generic and don't necessarily have the specialization that we need going forward. For example, files() doesn't distinguish between exclusive storage and shared storage. This makes stream clone difficult to implement when e.g. there may be a single file backing storage for multiple tracked paths. It also makes reporting difficult, as we don't know how many bytes are actually used by storage since we can't easily identify shared files. This commit implements a new method for obtaining storage metadata. It is designed to accept arguments specifying what metadata to request and to return a dict with those fields populated. We /could/ make each of these attributes a separate method. But this is a specialized API and I'm trying to avoid method bloat on the interfaces. There is also the possibility that certain callers will want to obtain multiple fields in different combinations and some backends may have performance issues obtaining all that data via separate method calls. Simple storage integration tests have been added. For now, we assume fields can't be "None" (ignoring the interface documentation). We can revisit this later. Differential Revision: https://phab.mercurial-scm.org/D4747

lfs: drop unused import A recent change dropped the last user of this module. Differential Revision: https://phab.mercurial-scm.org/D4744