narrow: pass old includes and excludes to _widen() In future patches we will need to pass them in the widen wireprotocol command which we are building. Differential Revision: https://phab.mercurial-scm.org/D4812

narrow: check for servers' narrow support before doing anything (BC) Recently we introduced narrow capabilities for the server. So we can check whether a server has narrow clone support or not before doing anything. This is BC because new clients won't be able to extend from old narrow-enabled servers. I *think* narrow is not used much (maybe just inside Google), also it's experimental so I think we can change this. We will need to this someday anyway. The "doesn't" in error is changed to "does not" because I think that's we do in core. I also changed one more instance of the error message to use 'does not' for consistency. Differential Revision: https://phab.mercurial-scm.org/D4789

narrow: don't do the dirstate dance if ellipses is not enabled I believe we set dirstate parents to nullid before widening pull because in ellipses cases, the parent might be stripped off with a new changeset. However the second ds.setparents() call invalidate my assumption. I am not sure why we do this. So here is a patch. This patch also adds tests showing we break nothing in non-ellipses cases. Differential Revision: https://phab.mercurial-scm.org/D4788

narrow: pass 'narrow_widen' as source while generating changegroup Extensions inspect the `source` parameter and add some custom logic on the basis of value of that parameter. The parameter is also passed to hooks. So let's pass the value as 'narrow_widen' to differentiate widening from pull. Differential Revision: https://phab.mercurial-scm.org/D4787

narrow: factor out logic to create cg while widening into separate fn This patch takes out the logic which generates a changegroup for widening a narrow clone when ellipses are disabled. This is done because future patches will introduce a narrow_widen() wireprotocol command which will send a bundle2 with changegroup and will use this function. The new function for now returns just the changegroup for compatibility with existing code, but in future patches, when we establish a wireprotocol command and call this function from there, this will return the required bundle2. Differential Revision: https://phab.mercurial-scm.org/D4786

narrow: avoid looking up dirstate again when editing dirstate The narrow extension overrides the dirstate editing functions to restrict paths outside the narrowspec. These overrides access the dirstate from repo.dirstate instead of using the "self" reference passed to the overridden functions. I don't see a reason for this and it caused me problems with a later patch (it caused infinite recursion when I modified localrepo.dirstate()), so let's change it. Differential Revision: https://phab.mercurial-scm.org/D4829

repo: move unfiltered-repo optimization to workingctx localrepo.__getitem__ special-cased lookup of the working copy parent to avoid looking up obsmarkers. I think the reason for that code (which I once wrote myself) was to make `hg commit` not load obsmarkers, which it would otherwise do via ctx.p1() in localrepo.commitctx(). That had the somewhat unfortunate consequence of making lookup of an unrelated binary nodeid load the dirstate. Now that changectx's constructor is dumb, we can let workingctx._parents() have the opmtimization instead. This affects two tests, because they no longer end up loading the dirstate and their "warning: ignoring unknown working parent ..." messages therefore go way. Differential Revision: https://phab.mercurial-scm.org/D4828

context: move logic from changectx.__init__ to localrepo.__getitem__ (API) My motivation for this change was to make repo[node] not load the dirstate (more about that in the next patch), but I think it makes more sense this way too. For example, raising RepoLookupError seems to belong better in the repo lookup function (i.e. localrepo.__getitem__). This makes the changectx constructor very simple -- it just assigns the given repo, revnum, and nodeid to properties. Differential Revision: https://phab.mercurial-scm.org/D4827

context: reduce dependence of changectx constructor I want to change the constructor's signature and letting all creation of changectx instances go through the repo simplifies that. Differential Revision: https://phab.mercurial-scm.org/D4826

py3: use system strings for HTTP response header comparison res.headers is holding system strings. Attempting to compare against bytes always fails and we fail to print decoded CBOR responses on Python 3. After this change, various test-wireproto* tests are now properly printing CBOR response objects. Differential Revision: https://phab.mercurial-scm.org/D4835

py3: convert HTTP request headers to str The low-level request object ideally takes system strings for HTTP request headers and values. If we send in bytes, it works. But a duplicate header check fails and various tests emit duplicate user-agent headers. Differential Revision: https://phab.mercurial-scm.org/D4834

py3: use system strings in HTTP server code Previously the source transformer was converting some string literals to bytes and we were comparing a system native string to bytes and this was leading to sending the wrong HTTP response headers on Python 3. After this change, we now properly send Transfer-Encoding and Connection response headers on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4833

py3: byteify test-storage.py The test now passed on Python 3. # skip-blame just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4836

narrow: remove narrowpatch instead of narrowcopies 84092edd5c88 removed narrowpatch.py and wanted to drop narrowpatch from __init__.py but mistakenly removed narrowcopies. This will make buildbots green again. Differential Revision: https://phab.mercurial-scm.org/D4837

py3: suppress the output from .write() calls in more tests I missed these in 803b7569c9ea because I forgot about `...` lines.

py3: byteify test-lfs.t

lfs: explicitly name a key when sorting blob pointers This is needed to keep py3 happy. The other two instances of sorting already did this.

py3: byteify test-lock.py There's still a stacktrace about not being able to serialize _io.TextIOWrapper.

py3: byteify contrib/dumprevlog

py3: quote $PYTHON in test-merge-symlinks.t

copies: inline a variable that's used only once By inlining it, we also avoid calculating the value when "if of in seen" is false (probably not significant). Differential Revision: https://phab.mercurial-scm.org/D4831

bundlerepo: remove a variable alias "parentrepo" and "repo" were the same thing and I don't see much reason for it (unionrepo has similar structure and a similar alias but there are two repos there so at least it makes a little more sense there). Differential Revision: https://phab.mercurial-scm.org/D4830

merge with stable

narrow: move support for `hg verify` into core Differential Revision: https://phab.mercurial-scm.org/D4824

narrow: drop unnecessary overrides of patch The patch overrides became unnecessary when I made context.status() filter by the narrowspec in e411774a2e0f (narrow: move status-filtering to core and to ctx, 2018-08-02). Differential Revision: https://phab.mercurial-scm.org/D4823

tests: add test showing that paths outside narrowspec are not added We had a test that shows that trying to explicitly add a file outside the narrowspec results in an error, but we didn't have a test that shows that paths from subdirectories outside the narrowspec are not added (on e.g. `hg add .`) but silently ignored. Differential Revision: https://phab.mercurial-scm.org/D4822

catapipe: fix bug introduced when switching to the timer api Differential Revision: https://phab.mercurial-scm.org/D4818

narrow: move the ellipses server capability to core This will be used in core logic to determining whether a server is ellipses enabled or not. And also this will ease moving narrow related things to core. Differential Revision: https://phab.mercurial-scm.org/D4809

narrow: move the wireprotocol narrow capability name to core We are trying to integrate the whole of narrow logic into core and it will be helpful for upcoming patches to have these capability names in core. The next patch will move the ellipses capability to core also. The exact motivation is to know whether we are cloning a ellipses repo or not and adding an ellipses repo requirement. Differential Revision: https://phab.mercurial-scm.org/D4808

help: document about "files" template keywords

formatter: remove experimental marker from -T option I think it's good enough to stabilize the feature, at least for the core and non-experimental commands.

grep: rename {line_number} to {lineno} as well (BC)

annotate: rename {line_number} to {lineno} (BC) I think {lineno} looks more like a common template keyword. It isn't called a {line} to avoid conflicts with the element name of {lines} and the {_|splitlines} filter. https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Dictionary

resolve: rename {status} to {mergestatus} to not shadow change status (BC) This is a part of the name unification. I think it's somewhat useful to provide a combined view of change/merge statuses. https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Dictionary

py3: whitelist one additional test Differential Revision: https://phab.mercurial-scm.org/D4817

logtoprocess: connect all fds to /dev/null to avoid bad interaction with pager We detected that pager is waiting for log-to-process script to finish, which is annoying when adding a script on commandfinish that does an HTTP push. There seems to be no workaround on the script side and it will make the behavior on Linux/MacOS closer to the Windows behavior. The drawback is that it makes the related tests more flaky as log-to-process outputs are now really asynchronous. If it's considered a BC change, another option would be to add a config option for this new behavior. I personally think that the different behavior between Windows and Linux is confusing and that it's a bug I would be fine with a new config option. Differential Revision: https://phab.mercurial-scm.org/D4816

logtoprocess: add a test to show pager and ltp bad interaction The next changeset will fix the bad interaction between the pager and log-to- process. Add a test first to make the next changeset easier to understand. Differential Revision: https://phab.mercurial-scm.org/D4815

py3: add one more passing test to whitelist caught by buildbot Thanks to Matt Harbison who fixed the remaining failures of this test. Differential Revision: https://phab.mercurial-scm.org/D4814

cleanupnodes: pass multiple predecessors to `createmarkers` directly

obsolete: allow multiple predecessors in createmarkers The logic for this change is similar to the change to `cleanupnodes` that we did earlier. Now that the rebase code is trying to record a fold, we need to actually record it in the markers. The first step is to have the markers creation API able to receive such fold data. To keep things sane, we restrict fold to on successors.

obsolete: preindent code in createmarkers This will help to make the next change clearer.

obsolete: explicitly pass relation items to effectflag computation To track folds, we are about to change the content of `rel`. To simplify this change, we update the `geteffectflag` function beforehand.

rebase: explicitly track collapses as fold A collapse is a large fold. It is now explicitly tracked (at the cleanupnodes API level).

rebase: use tuple as `replacement` keys Now that `cleanupnodes` support tuples as key, we update the rebase code to use them. No changes in the replacement tracked are introduced yet.

rebase: expand a long "one-liner" When a one-liner gets 3 lines longs, it lose its expressivity benefits. We expand it into a simple for loop. This makes future changes of the code in that area clearer.

cleanupnodes: drop special casing around prune markers (API) The `cleanupnodes` has logic to skip the creation of "prune" markers if the changeset is already obsolete. This feels strange and gets in the way of code changes to tracks folds. Now that callers no longer request such prune, we can drop this logic. In many cases, pruning through cleanupnodes should be replaced by internal phase usage.

rebase: don't try to prune obsolete changeset already in the destination With similar motivations to the previous changesets, we stop marking changeset from pruning when it is not the command intention. In this case, we still need to distinguish between the strip and the obsolete case.

histedit: don't cleanup nodes already disposed of If something else took care of these temporary nodes, we don't need to do anything about it. This less liberal usage of pruning through cleanup nodes will help us further cleanup on the road to explicitly tracks folds.

py3: avoid b'' output in test-hgweb-non-interactive.t I couldn't figure out how to get the list to print without b'' with pycompat.write(), without converted each element to str.

py3: convert print to a function call in a few tests I *think* this is the last of them.

py3: use util.forcebytestr() to convert push lock error to bytes

py3: byteify windows.shelltocmdexe() This makes test-doctest.py happy on Windows.

run-tests: flush output stream before prompting to accept changes With py3 on Windows, the prompt to accept changes in an --interactive run wasn't showing up until after the choice was read from stdin.

py3: use util.forcebytestr() to convert error messages to bytes This makes the python 3 buildbot green again. Differential Revision: https://phab.mercurial-scm.org/D4811

context: stop catching TypeError when converting hex nodeid to binary It has been a programming error to pass a 40-character string that is not a hex nodeid since 8b86acc7aa64 (context: drop support for looking up context by ambiguous changeid (API), 2018-04-28), so we can just let it raise a TypeError. Differential Revision: https://phab.mercurial-scm.org/D4807

context: stop catching and re-raising FilteredRepoLookupError FilteredRepoLookupError is only raised by changectx's constructor and the higher-level scmutil.revsymbol(), so there's no need to catch it in changectx's constructor. Differential Revision: https://phab.mercurial-scm.org/D4806

util: use a context manager in readlock()

py3: convert os.readlink() path to native strings on Windows Windows insisted that it needs to be str. I skipped the stuff in the posix module, and left `tests/f` and `run-tests.py` alone for now.

py3: suppress the output from .write() calls in the remaining tests

py3: conditionalize json float precision difference in test-debugcommands.t

py3: byteify extension in test-debugcommands.t

pullreport: rev duplicated and extinct into account If we already have some obsolete and hidden nodes locally and the server send them again to you, it seems useful to point it out instead of being silent about it.

pullreport: issue a message about "extinct" pulled changesets Changeset pulled from a remote repository while already obsolete locally can end up hidden after the pull. Hiding obsolete changesets is a good behavior but silently "skipping" some of the pulled content can get confusing. We now detect this situation and emit a message about it. The message is simple and the wording could be improved, however, we focus on the detection here. Evolution is still an experimental feature, so the output is open to changes. In particular, we could point out at the latest successors of the obsolete changesets, however, it can get tricky is there are many of them. So we delay these improvements to another adventure. Another easy improvement would be to merge this message with the previous line about the new nodes and their phases. This is a good example of cases where we can only transmit a limited amount of data to users by default. We need some sort of "transaction journal" we could point the user to.

pullreport: skip or rework some early return We are about to add more logic in this report. Before that, we need it to not quit so early.

pullreport: skip filtered revs instead of obsolete ones Obsolescence is closely related to visibility but still a distinct concept. We can receive changesets that are obsolete but visible (eg: when pulling orphans). Such changeset should be reported too. In addition, the filtering level can be anything, we should respect it.

pullreport: add a test to show misreporting of visible changeset The current code ignores all obsolete changesets including the visible one. We add a test showing this behavior before fixing the behavior.

repo: don't look up context for tip node if it's not needed We were doing repo['tip'].node() or similar in a few places where repo.changelog.tip() would be enough. Differential Revision: https://phab.mercurial-scm.org/D4781

repo: look up nullrev context by revnum, not symbolic name I think lookup of the 'null' symbol should be done via scmutil.revsymbol() and repo['null'] is only supported for historical reasons. However, repo[nullrev] is fine, so we can switch to that instead of switching to scmutil.revsymbol('null'). Differential Revision: https://phab.mercurial-scm.org/D4780

bundle: consistently put revnums in "base" collection The "base" collection contains revnums, except that it can also contain the "null" symbol. That's a little weird. Let's be consistent. Differential Revision: https://phab.mercurial-scm.org/D4779

py3: ensure printing to stdout uses str in test-hgweb-no-request-uri.t

scmutil: accept multiple predecessors in 'replacements' (API) This changeset makes 'cleanupnodes' accepts multiple predecessors as `replacements` keys. The same as it accepts multiple successors as `replacements` values. To avoid breaking all callers, the old and new ways are currently valid at the same time. We'll deprecate and drop the old way later. This change is the first step toward a better tracking of "fold" event in the evolution history. While working on the "rewind" command (in the evolve extension), we realized that first class tracking of folds are necessary. We already have good tracking of splits. When walking the evolution history from predecessors to successors, that makes for a clear distinction between having multiple successors because of the actual splitting of a changeset or content-divergences. The "rewind" command allows restoring older evolution of a stack of changesets. One of its mode walks the evolution history to automatically find appropriate predecessors. This means walking from successors to predecessors. In this case, we need to be able to make the same distinction between an actual fold and other cases. So we will have to track folds explicitly. This changesets only focus on making it possible to express fold at the `cleanupnodes` API level. The actual tracking will be implemented later.

scmutil: expand long "one-liner" When a one-liner gets 3 lines longs, it loses its expressivity benefits. We expand it into a simple for loop. This makes future changes of the code in that area clearer.

shelve: find shelvedctx from bundle even if they are already in the repo We use the new "duplicates" node tracking to find the tip of the bundle even if it already exists in the repository. Such logic is not supposed to be needed in theory. If the shelve was made using internal-phase, we already know its node. Otherwise, the bundle content should have been stripped. However, handling it makes the shelve code more robust and provide a good example of "revduplicates" usage.

shelve: return the shelved node as part of bundle application It make sense to have the function in charge of unbundling the shelved revision also return the node of that revision (when the data is in the bundle). This will help us to handle unnatural state where the unshelved change already exists in the repository.

changelog: keep track of duplicated node in the transaction adding them The transaction is already tracking the new nodes. We now tracks the "duplicates" in the same location.

revlog: add a callback "tracking" duplicate node addition If a changegroup contains node already added to the repository, they will be skipped. Skipping them is the right behavior (we don't need to store things twice), but it can hide some information to the code doing the unbundle (eg: shelve looking for the tip of the bundle). The first step to improve this situation is to add a low level callback. We do not need this tracking on all revlog, so actual tracking will be added in the next changeset.

logtoprocess: define $HG for children processes So they can compute the hg version for instance. Differential Revision: https://phab.mercurial-scm.org/D4768

py3: mask out None type when printing in `debuglocks` Apparently, %b doesn't allow None.

py3: ensure standard exceptions use `str` type strings in windows.py See also edaa40dc5fe5.

py3: replace a StandardError reference This doesn't exist on py3, and the standard way of handling this seems to be to catch both exceptions.

storageutil: extract revision number iteration This code is a bit quirky (and possibly buggy). It will likely be used by multiple storage backends. Let's extract it so it is reusable. Differential Revision: https://phab.mercurial-scm.org/D4757

storageutil: new function for extracting metadata-less content from text Other storage backends will want to do this. I'm not concerned about Python function call overhead because I expect self.revision() to dwarf the function call overhead time, since self.revision() requires multiple function calls and may involve decompression in the common case. Differential Revision: https://phab.mercurial-scm.org/D4756

storageutil: move _censoredtext() from revlog This seems like generic functionality we'll want to use from non-revlog storage backends. Differential Revision: https://phab.mercurial-scm.org/D4755

storageutil: move metadata parsing and packing from revlog (API) Parsing and writing of revision text metadata is likely identical across storage backends. Let's move the code out of revlog so we don't need to import the revlog module in order to use it. Differential Revision: https://phab.mercurial-scm.org/D4754

storageutil: new module for storage primitives (API) There will exist common code between storage backends. It would be nice to have a central place to put that code. This commit attempts to create that place by creating the "storageutil" module. The first thing we move is revlog.hash(), which is the function for computing the SHA-1 hash of revision fulltext and parents. Differential Revision: https://phab.mercurial-scm.org/D4753

filelog: stop proxying deltaparent() (API) deltaparent() obtains the revision number of the base revision a delta in storage is stored against. It is highly revlog-centric and may not apply to other storage backends. As a result, it doesn't belong on the generic file storage interface. This method/proxy is no longer used in core. The last consumer was probably changegroup code and went away with the transition to emitrevisions(). Differential Revision: https://phab.mercurial-scm.org/D4751

filelog: stop proxying rawsize() (API) This method is no longer used by external consumers. The API is quite low-level and is effectively len(revision(raw=True)). I don't see a compelling reason to keep it around. Let's drop the API and make the file storage interface simpler. Differential Revision: https://phab.mercurial-scm.org/D4750

filelog: stop proxying "opener" (API) The last consumer of it in upgrade code was removed as part of the previous commit. This attribute is revlog specific (because it assumes the existence of a vfs for performing I/O on tracked file data) and therefore isn't appropriate for a generic storage interface. So nuke it. Differential Revision: https://phab.mercurial-scm.org/D4749

filelog: stop proxying flags() (API) Per-revision storage flags are kinda a revlog-centric API. (Except for the fact that changegroup uses the same integer flags as revlog does and there's minimal verification that the server's flags map to the client's storage flags - but that's another problem.) The last user of flags() was in verify.py and that code was just moved into revlog.py and is accessed behind the verifyintegrity() file storage API. Since there are no more consumers, let's drop the proxy and remove the method from the file storage interface. This commit only drops the dedicated API for reading a single revision's storage flags: we still support reading and writing flags through the bulk data retrieval and add revision APIs. And since changegroups encode revlog integer flags over the wire, we'll always need to support flags at some level. The removal of individual storage flags may be too premature. But since flags() is now unused, I'd like to see how far we can get without that dedicated API - especially since it uses revision numbers instead of nodes. Differential Revision: https://phab.mercurial-scm.org/D4746

revlog: move revision verification out of verify File revision verification is performing low-level checks of file storage, namely that flags are appropriate and revision data can be resolved. Since these checks are somewhat revlog-specific and may not be appropriate for alternate storage backends, this commit moves those checks from verify.py to revlog.py. Because we're now emitting warnings/errors that apply to specific revisions, we taught the iverifyproblem interface to expose the problematic node and to report this node in verify output. This was necessary to prevent unwanted test changes. After this change, revlog.verifyintegrity() and file verify code in verify.py both iterate over revisions and resolve their fulltext. But they do so in separate loops. (verify.py needs to resolve fulltexts as part of calling renamed() - at least when using revlogs.) This should add overhead. But on the mozilla-unified repo: $ hg verify before: time: real 700.640 secs (user 585.520+0.000 sys 23.480+0.000) after: time: real 682.380 secs (user 570.370+0.000 sys 22.240+0.000) I'm not sure what's going on. Maybe avoiding the filelog attribute proxies shaved off enough time to offset the losses? Maybe fulltext resolution has less overhead than I thought? I've left a comment indicating the potential for optimization. But because it doesn't produce a performance regression on a large repository, I'm not going to worry about it. Differential Revision: https://phab.mercurial-scm.org/D4745

tests: de-flake test-narrow-debugrebuilddirstate.t If the dirstate gets written much later (usually 1-2 s, depending on FS) than the working copy file (there's only one), then the `hg debugdirstate` command will include a timestamp. There's nothing wrong with that, so we should just allow it. Differential Revision: https://phab.mercurial-scm.org/D4758

upgrade: use storageinfo() for obtaining storage metadata Let's switch to our new API for obtaining information about storage. This eliminates the last consumer of rawsize() and the opener proxy from the file storage interface! Differential Revision: https://phab.mercurial-scm.org/D4748

revlog: add method for obtaining storage info (API) We currently have a handful of methods on the file and manifest storage interfaces for obtaining metadata about storage. e.g. files() is used to obtain the files backing storage. rawsize() is to quickly compute the size of tracked revisions without resolving their fulltext. Code in upgrade and stream clone make heavy use of these methods. The existing APIs are generic and don't necessarily have the specialization that we need going forward. For example, files() doesn't distinguish between exclusive storage and shared storage. This makes stream clone difficult to implement when e.g. there may be a single file backing storage for multiple tracked paths. It also makes reporting difficult, as we don't know how many bytes are actually used by storage since we can't easily identify shared files. This commit implements a new method for obtaining storage metadata. It is designed to accept arguments specifying what metadata to request and to return a dict with those fields populated. We /could/ make each of these attributes a separate method. But this is a specialized API and I'm trying to avoid method bloat on the interfaces. There is also the possibility that certain callers will want to obtain multiple fields in different combinations and some backends may have performance issues obtaining all that data via separate method calls. Simple storage integration tests have been added. For now, we assume fields can't be "None" (ignoring the interface documentation). We can revisit this later. Differential Revision: https://phab.mercurial-scm.org/D4747

lfs: drop unused import A recent change dropped the last user of this module. Differential Revision: https://phab.mercurial-scm.org/D4744

filelog: drop _generaldelta attribute (API) With changegroup moving to emitrevisions(), this revlog-specific attribute is no longer used and can be deleted. Good riddance. Differential Revision: https://phab.mercurial-scm.org/D4727

revlog: drop emitrevisiondeltas() and associated functionality (API) emitrevisions() is the future! Differential Revision: https://phab.mercurial-scm.org/D4726