storageutil: extract copy metadata retrieval out of filelog As part of implementing an alternate storage backend, I found myself reinventing this wheel. Let's create a utility function for doing the work. Differential Revision: https://phab.mercurial-scm.org/D4800

storageutil: extract functionality for resolving strip revisions I found myself having to copy this method as part of implementing a new storage backend. With a little tweaking, we can extract it to a standalone function so it can be reused easily. We'll likely want to implement a better method for removing revisions on the storage interface someday. But until then, let's use what we have. Differential Revision: https://phab.mercurial-scm.org/D4799

storageutil: consistently raise LookupError (API) The interface docs say this is supposed to raise LookupError on failure. But for invalid revision number input, it could raise IndexError because ifileindex.node() is documented to raise IndexError. lookup() for files isn't used that much (pretty much just in basefilectx in core AFAICT). And callers should already be catching LookupError. So I don't anticipate that much fallout from this change. Differential Revision: https://phab.mercurial-scm.org/D4798

storageutil: implement file identifier resolution method (BC) revlog.lookup() has a number of advanced features, including partial node matching. These advanced features aren't needed for file id lookup because file identifiers are almost always from internal sources. (An exception to this is hgweb, which appears to have some code paths that attempt to resolve a user-supplied string to a node.) This commit implements a new function for resolving file identifiers to nodes. It behaves like revlog.lookup() but without the advanced features. Tests reveal behavior changes: * Partial hex nodes are no longer resolved to nodes. * "-1" now returns nullid instead of raising LookupError. * "0" on an empty store now raises LookupError instead of returning nullid. I'm not sure why "-1" wasn't recognized before. But it seems reasonable to accept it as a value since integer -1 resolves to nullid. These changes all seem reasonable to me. And with the exception of partial hex node matching, we may want to consider changing revlog.lookup() as well. Differential Revision: https://phab.mercurial-scm.org/D4797

testing: add more testing for ifileindex.lookup() The tests demonstrate some... questionable behavior of revlog.lookup(). Differential Revision: https://phab.mercurial-scm.org/D4796

dagop: extract DAG local heads functionality from revlog As part of implementing an alternate storage backend, I found myself having to reimplement this function. The DAG traversal logic is generic and can be factored out into a standalone function. We could probably combine this with headrevs(). But I'll leave that for another time. Differential Revision: https://phab.mercurial-scm.org/D4795

dagop: extract descendants() from revlog module This method needs to be implemented in other storage backends and is generic if we parameterize the functions for retrieving revision numbers and parent revision numbers. Let's extract it to dagop so backends don't need to reinvent the wheel. I'm not too worried about performance overhead of the extra function call because I don't expect anyone to be calling descendants() in a tight loop. Differential Revision: https://phab.mercurial-scm.org/D4794

filelog: remove checkhash() (API) It is unused. While a caller may want to ask the store to validate the hash of some provided text, since there are no in-core consumers of this method, let's drop it for now. Differential Revision: https://phab.mercurial-scm.org/D4793

filelog: remove revdiff() (API) This proxy method is no longer used. While it might be useful to query a storage backend for the delta between any 2 revisions because the store could have a delta cached and could compute it more efficiently than the caller calling revision() twice in order to compute a delta, since nothing in core is using this API now, I feel comfortable nuking it. Differential Revision: https://phab.mercurial-scm.org/D4792

localrepo: define storage backend in creation options (API) We add an experimental config option to define the storage backend for new repositories. By default, it uses "revlogv1," which maps to the current and only modern supported repository format. We add a "backend" creation option to control which backend to use. It defaults to using the value from the config option. newreporequirements() will now barf if it sees a "backend" value that isn't "revlogv1." This forces extensions to monkeypatch the function to handle requirements derivation for custom backends. In order for this to "just work," we factored out obtaining the default creation options into its own function and made callers of newreporequirements() responsible for passing in valid data. Without this, direct callers of newreporequirements() wouldn't get the proper results. Differential Revision: https://phab.mercurial-scm.org/D4791

wireprotov2: derive "required" from presence of default value If we define a default value for all optional arguments, we no longer need to explicitly declare whether the argument is required. Instead, we can derive it from the presence of a default value callable. Differential Revision: https://phab.mercurial-scm.org/D4790

localrepo: capture repo interface factory functions as lambas Previously, the list would hold a reference to the original function and attempting to wrap the module function via extension wouldn't result in the wrapped function being called, completely undermining the expected behavior. Differential Revision: https://phab.mercurial-scm.org/D4821

extensions: new closehead module for closing arbitrary heads ``hg close-head`` allows closing arbitrary heads. It is equivalent to checking out the given revisions and commit an empty change with ``hg commit --close-branch``. Differential Revision: https://phab.mercurial-scm.org/D3557

cext: use modern buffer protocol in mpatch_flist() Differential Revision: https://phab.mercurial-scm.org/D4841

cext: use modern buffer protocol in patches() PyObject_AsCharBuffer() is part of the "Old Buffer Protocol," which has been deprecated for years. Let's port away from it. PyBuffer_GetBuffer() must be paired with PyBuffer_Release(), hence the added "goto cleanup" in a failure case. We don't bump the extension version because the API has not changed. Differential Revision: https://phab.mercurial-scm.org/D4840

identify: when using -T, avoid unnecessary remote bookmarks query Differential Revision: https://phab.mercurial-scm.org/D4839

identify: only query remote bookmarks if needed Instead of all the time when operating on a remote repo. This perf regression was introduced in 15a79ac823e8, in 4.3. This datahint method returns nothing for -Tjson, -Tpickle, -Tdebug --config ui.formatdebug=true and --config ui.formatjson, so the bookmarks won't show up. I don't know what these formatters are for. plainformatter and templateformatter work properly, and the few other uses of datahint should have the same kind of problem. There is further weirdness where "--template '{node}'" is not enough to avoid querying the bookmarks, you also need to pass --id or -q. Differential Revision: https://phab.mercurial-scm.org/D4819

py3: whitelist another passing tests caught by buildbot I this indygreg recent patches made this test pass on Python 3. So thanks to him! Differential Revision: https://phab.mercurial-scm.org/D4848

manifest: remove an unused variable caught by pyflakes Uses of todel were removed in D4843 (19103e68a698). This patch will make buildbots green again. Differential Revision: https://phab.mercurial-scm.org/D4847

setup: ignore message about disabling 3rd party extensions because of version I started getting into a bind recently when switching between py2 and py3 because switching requires a `make clean`, which kills __version__.py. But then when running `make local`, it picks up the local hg.exe (MSYS seems to prefix $PATH with '.'), which doesn't know its version. That causes it to emit a warning about needing at least 4.3 to load evolve, which caused setup.py to fail saying there is no working hg executable to figure out the version. If we can ignore general extension import failures, we should be able to ignore this too.

narrow: avoid overwriting a variable I don't like to overwrite variables, especially not with a different type (because it makes it harder to explain what the variable is for, and, therefore, to name it well). Differential Revision: https://phab.mercurial-scm.org/D4846

treemanifests: remove _loadalllazy in _diff() The benchmarks below use a similar setup as in ee7ee0c516ca and my other recent commits. Yes, in some cases this runs in literally 5% of the time it previously took. before = a6f8ab53 diff --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.301 s +- 0.004 s | 1.309 s +- 0.012 s | 100.6% m-u | | x | 1.303 s +- 0.009 s | 1.302 s +- 0.006 s | 99.9% m-u | x | | 1.308 s +- 0.006 s | 1.309 s +- 0.007 s | 100.1% m-u | x | x | 85.7 ms +- 0.6 ms | 86.0 ms +- 0.3 ms | 100.4% l-d-r | | | 197.5 ms +- 0.7 ms | 197.8 ms +- 2.2 ms | 100.2% l-d-r | | x | 199.4 ms +- 0.6 ms | 199.3 ms +- 0.9 ms | 99.9% l-d-r | x | | 86.1 ms +- 0.5 ms | 85.8 ms +- 0.9 ms | 99.7% l-d-r | x | x | 64.4 ms +- 0.4 ms | 64.4 ms +- 0.3 ms | 100.0% diff -c . --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 236.7 ms +- 1.1 ms | 236.5 ms +- 1.3 ms | 99.9% m-u | | x | 158.7 ms +- 1.0 ms | 128.0 ms +- 1.0 ms | 80.7% <-- m-u | x | | 239.7 ms +- 1.8 ms | 238.1 ms +- 1.5 ms | 99.3% m-u | x | x | 132.4 ms +- 0.9 ms | 132.3 ms +- 0.6 ms | 99.9% l-d-r | | | 81.8 ms +- 0.4 ms | 81.8 ms +- 0.3 ms | 100.0% l-d-r | | x | 3.894 s +- 0.017 s | 193.6 ms +- 0.4 ms | 5.0% <-- l-d-r | x | | 106.9 ms +- 0.4 ms | 106.6 ms +- 0.3 ms | 99.7% l-d-r | x | x | 182.7 ms +- 0.8 ms | 183.3 ms +- 0.9 ms | 100.3% rebase -r . --keep -d .^^: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 5.615 s +- 0.109 s | 5.562 s +- 0.015 s | 99.1% m-u | | x | 5.701 s +- 0.027 s | 5.715 s +- 0.023 s | 100.2% m-u | x | | 5.572 s +- 0.128 s | 5.613 s +- 0.182 s | 100.7% m-u | x | x | 633.3 ms +- 28.7 ms | 636.2 ms +- 13.8 ms | 100.5% l-d-r | | | 666.4 ms +- 17.0 ms | 658.5 ms +- 9.3 ms | 98.8% l-d-r | | x | 6.520 s +- 0.070 s | 6.505 s +- 0.026 s | 99.8% l-d-r | x | | 279.0 ms +- 13.0 ms | 276.5 ms +- 4.7 ms | 99.1% l-d-r | x | x | 1.636 s +- 0.058 s | 1.657 s +- 0.014 s | 101.3% status --change . --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 218.6 ms +- 1.4 ms | 217.9 ms +- 1.5 ms | 99.7% m-u | | x | 138.5 ms +- 0.4 ms | 108.4 ms +- 0.2 ms | 78.3% <-- m-u | x | | 220.1 ms +- 1.3 ms | 219.7 ms +- 1.5 ms | 99.8% m-u | x | x | 113.2 ms +- 0.4 ms | 112.4 ms +- 0.8 ms | 99.3% l-d-r | | | 80.2 ms +- 0.3 ms | 80.6 ms +- 0.6 ms | 100.5% l-d-r | | x | 3.899 s +- 0.020 s | 194.8 ms +- 4.0 ms | 5.0% <-- l-d-r | x | | 83.4 ms +- 0.8 ms | 83.2 ms +- 0.2 ms | 99.8% l-d-r | x | x | 732.2 ms +- 4.3 ms | 194.9 ms +- 1.0 ms | 26.6% <-- status --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.917 s +- 0.005 s | 1.914 s +- 0.004 s | 99.8% m-u | | x | 1.909 s +- 0.012 s | 1.934 s +- 0.004 s | 101.3% m-u | x | | 1.915 s +- 0.005 s | 1.904 s +- 0.004 s | 99.4% m-u | x | x | 94.8 ms +- 0.3 ms | 94.7 ms +- 0.2 ms | 99.9% l-d-r | | | 593.9 ms +- 1.2 ms | 594.6 ms +- 9.4 ms | 100.1% l-d-r | | x | 595.2 ms +- 3.8 ms | 597.2 ms +- 2.6 ms | 100.3% l-d-r | x | | 182.5 ms +- 1.6 ms | 182.1 ms +- 0.6 ms | 99.8% l-d-r | x | x | 149.6 ms +- 0.9 ms | 149.1 ms +- 0.8 ms | 99.7% update $rev^; ~/src/hg/hg{hg}/hg update $rev: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 3.121 s +- 0.007 s | 3.129 s +- 0.012 s | 100.3% m-u | | x | 2.972 s +- 0.011 s | 2.981 s +- 0.012 s | 100.3% m-u | x | | 3.144 s +- 0.014 s | 3.141 s +- 0.011 s | 99.9% m-u | x | x | 312.2 ms +- 2.4 ms | 312.3 ms +- 2.1 ms | 100.0% l-d-r | | | 444.4 ms +- 4.3 ms | 446.9 ms +- 5.3 ms | 100.6% l-d-r | | x | 9.159 s +- 0.069 s | 9.182 s +- 0.040 s | 100.3% l-d-r | x | | 254.6 ms +- 1.6 ms | 255.2 ms +- 1.6 ms | 100.2% l-d-r | x | x | 1.525 s +- 0.007 s | 1.577 s +- 0.007 s | 103.4% <--? Differential Revision: https://phab.mercurial-scm.org/D4845

treemanifests: skip extraneous check for item before calling _loadlazy Differential Revision: https://phab.mercurial-scm.org/D4844

treemanifests: make _loadchildrensetlazy just call _loadlazy This was suggested on the original review for _loadchildrensetlazy (D4370), it's taken me a while to get to it though. Differential Revision: https://phab.mercurial-scm.org/D4843

treemanifests: make _loadlazy tolerate item not on _lazydirs I'd like to clean up a few cases where we check for an item in _lazydirs before calling _loadlazy - this will remove an extraneous dict lookup and make it slightly more versatile. Differential Revision: https://phab.mercurial-scm.org/D4842

debugcommands: add a debugindexstats command Someone went through the trouble of recording these stats, so let's make them accessible. Differential Revision: https://phab.mercurial-scm.org/D4832

rust-chg: add main program It can at least connect to a *running* command server, and execute Mercurial command. No signal handling nor daemon management is implemented yet.

rust-chg: add interface to chdir the server

rust-chg: add Client extensions to run cHg-specific requests This just provides a nicer way to issue command-server requests.

rust-chg: port basic socket path handling from cHg of C This is basically modeled after setcmdserveropts() of chg.c.

rust-chg: add state machine to handle "runcommand" request with cHg extension This is modeled after tokio-hglib's RunCommand state to support the "S" channel message.

rust-chg: add callback to handle pager and shell command requests This could be inlined into the ChgRunCommand state to be introduced by the next patch, but it seemed good to separate any user interactions from the IPC code.

rust-chg: add low-level function to set pager fd blocking This is necessary because the server expects stdout/stderr to be blocking, whereas we'll use async library to spawn pager, which makes pipes unblocking.

rust-chg: add future that handles "attachio" request This is the sequence to send client-side stdio and pager stdin to the server.

rust-chg: add parser for request messages sent to "S" channel The data structure is documented at chgserver.py:channeledsystem().

rust-chg: add wrapper around C function

rust-chg: add function to send fds via domain socket As a beginning, I wrote some C. It's extracted from attachio() of contrib/chg/hgclient.c. Maybe it could be rewritten in Rust by using the libc (and/or nix) crates, but doing that wouldn't be trivial as the code depends on CMSG_*() macros. IMO, using C is better here.

rust-chg: update dependencies

rust-chg: add project skeleton This directory will host the reimplementation of cHg in Rust. It will use Tokio [1], and tokio-hglib [2] which I wrote for an oxidized CHg, no idea if there's such carbon bonding in nature btw. [1]: https://tokio.rs/ [2]: https://bitbucket.org/yuja/tokio-hglib/ The reasoning for depending on Tokio is that it will allow us to handle Unix signals in a safer way. Well, I believed that until I found a weird function, handlestopsignal(), in cHg codebase. It resends the same signal to the same process by temporarily masking the handler, which can't be inherently async. So the signal handlers will stay in C, which means there isn't actually much reason to write async code right now, other than I've already done most of the async stuff, and slightly easier pager handling. The reasoning for the rewrite is that it will eventually be possible to port server-side config validation back to the client side, which will reduce the complexity of the current daemon management. It will also encourage us to write frontend library (e.g. command line and config parsers) in Rust. The license is GPL2+ because it's likely to include derived work from the cHg of C. The rust/chg crate is excluded from the root workspace as it's unclear how the whole rust packages should be laid out. That can be revisited later.

narrow: move copies overrides to core The copies overrides seems to have been a little complicated just by not being in core. When moved to core, it becomes trivial (at least I think these overrides have the same effect). Differential Revision: https://phab.mercurial-scm.org/D4825

narrow: pass old includes and excludes to _widen() In future patches we will need to pass them in the widen wireprotocol command which we are building. Differential Revision: https://phab.mercurial-scm.org/D4812

narrow: check for servers' narrow support before doing anything (BC) Recently we introduced narrow capabilities for the server. So we can check whether a server has narrow clone support or not before doing anything. This is BC because new clients won't be able to extend from old narrow-enabled servers. I *think* narrow is not used much (maybe just inside Google), also it's experimental so I think we can change this. We will need to this someday anyway. The "doesn't" in error is changed to "does not" because I think that's we do in core. I also changed one more instance of the error message to use 'does not' for consistency. Differential Revision: https://phab.mercurial-scm.org/D4789

narrow: don't do the dirstate dance if ellipses is not enabled I believe we set dirstate parents to nullid before widening pull because in ellipses cases, the parent might be stripped off with a new changeset. However the second ds.setparents() call invalidate my assumption. I am not sure why we do this. So here is a patch. This patch also adds tests showing we break nothing in non-ellipses cases. Differential Revision: https://phab.mercurial-scm.org/D4788

narrow: pass 'narrow_widen' as source while generating changegroup Extensions inspect the `source` parameter and add some custom logic on the basis of value of that parameter. The parameter is also passed to hooks. So let's pass the value as 'narrow_widen' to differentiate widening from pull. Differential Revision: https://phab.mercurial-scm.org/D4787

narrow: factor out logic to create cg while widening into separate fn This patch takes out the logic which generates a changegroup for widening a narrow clone when ellipses are disabled. This is done because future patches will introduce a narrow_widen() wireprotocol command which will send a bundle2 with changegroup and will use this function. The new function for now returns just the changegroup for compatibility with existing code, but in future patches, when we establish a wireprotocol command and call this function from there, this will return the required bundle2. Differential Revision: https://phab.mercurial-scm.org/D4786

narrow: avoid looking up dirstate again when editing dirstate The narrow extension overrides the dirstate editing functions to restrict paths outside the narrowspec. These overrides access the dirstate from repo.dirstate instead of using the "self" reference passed to the overridden functions. I don't see a reason for this and it caused me problems with a later patch (it caused infinite recursion when I modified localrepo.dirstate()), so let's change it. Differential Revision: https://phab.mercurial-scm.org/D4829

repo: move unfiltered-repo optimization to workingctx localrepo.__getitem__ special-cased lookup of the working copy parent to avoid looking up obsmarkers. I think the reason for that code (which I once wrote myself) was to make `hg commit` not load obsmarkers, which it would otherwise do via ctx.p1() in localrepo.commitctx(). That had the somewhat unfortunate consequence of making lookup of an unrelated binary nodeid load the dirstate. Now that changectx's constructor is dumb, we can let workingctx._parents() have the opmtimization instead. This affects two tests, because they no longer end up loading the dirstate and their "warning: ignoring unknown working parent ..." messages therefore go way. Differential Revision: https://phab.mercurial-scm.org/D4828

context: move logic from changectx.__init__ to localrepo.__getitem__ (API) My motivation for this change was to make repo[node] not load the dirstate (more about that in the next patch), but I think it makes more sense this way too. For example, raising RepoLookupError seems to belong better in the repo lookup function (i.e. localrepo.__getitem__). This makes the changectx constructor very simple -- it just assigns the given repo, revnum, and nodeid to properties. Differential Revision: https://phab.mercurial-scm.org/D4827