context: drop a redundant fast path in introrev Now that _adjustlinkrev fast path this case itself, we no longer need an extra conditional. A nice side effect is that we are no longer calling `self.rev()`. In case where `_descendantrev` is set, calling `self.rev` will trigger a potentially expensive `_adjustlinkrev` call. So blindly calling `self.rev()` to avoid another `_adjustlinkrev` call can be counterproductive. Note that `_descendantrev` is currently never taken into account in `introrev` so far which is wrong. We'll fix that in changeset later in this series.

context: fast path linkrev adjustement in trivial case If the search starts from the linkrev, there is nothing to adjust.

url: allow to configure timeout on http connection By default, httplib.HTTPConnection opens connection with no timeout. If the server is hanging, Mercurial will wait indefinitely. This may be an issue for automated scripts. Differential Revision: https://phab.mercurial-scm.org/D4878

obsolete: explicitly track folds inside the markers We now record information to be able to recognize "fold" event from obsolescence markers. To do so, we track the following pieces of information: a) a fold ID. Unique to that fold (per successor), b) the number of predecessors, c) the index of the predecessor in that fold. We will now be able to create an algorithm able to find "predecessorssets". We now store this data in the generic "metadata" field of the markers. Updating the format to have a more compact storage for this would be useful. This way of tracking a fold through multiple markers could be applied to split too. This would have two advantages: 1) We get a simpler format, since number of successors is limited to [0-1]. 2) We can better deal with situations where only some of the split successors are pushed to a remote repository. We should look into the relevance of such a change before updating the on-disk format. note: unlike splits, folds do not have to deal with cases where only some of the markers have been synchronized. As they all share the same successor changesets, they are all relevant to the same nodes.

cleanupnodes: update comment to drop mention of filtering Since changeset 1857f50a9643 drop the filtering, we should not longer mention it in code comment.

treemanifests: remove _loadalllazy when doing copies 'before' here is https://phab.mercurial-scm.org/D4845 (not the committed/rebased version) diff --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.329 s +- 0.011 s | 1.320 s +- 0.010 s | 99.3% m-u | | x | 1.316 s +- 0.005 s | 1.334 s +- 0.018 s | 101.4% m-u | x | | 1.330 s +- 0.021 s | 1.322 s +- 0.005 s | 99.4% m-u | x | x | 87.2 ms +- 0.7 ms | 86.9 ms +- 1.5 ms | 99.7% l-d-r | | | 203.3 ms +- 7.8 ms | 199.4 ms +- 1.8 ms | 98.1% l-d-r | | x | 204.6 ms +- 2.8 ms | 201.7 ms +- 2.1 ms | 98.6% l-d-r | x | | 90.5 ms +- 11.0 ms | 86.2 ms +- 1.0 ms | 95.2% l-d-r | x | x | 66.3 ms +- 2.0 ms | 66.4 ms +- 0.9 ms | 100.2% diff -c . --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 239.4 ms +- 2.0 ms | 241.7 ms +- 4.6 ms | 101.0% m-u | | x | 128.9 ms +- 1.9 ms | 130.9 ms +- 7.7 ms | 101.6% m-u | x | | 241.1 ms +- 1.6 ms | 240.1 ms +- 1.4 ms | 99.6% m-u | x | x | 133.4 ms +- 1.5 ms | 133.4 ms +- 1.2 ms | 100.0% l-d-r | | | 84.3 ms +- 1.5 ms | 83.5 ms +- 1.0 ms | 99.1% l-d-r | | x | 200.9 ms +- 6.3 ms | 203.0 ms +- 4.4 ms | 101.0% l-d-r | x | | 108.1 ms +- 1.4 ms | 108.7 ms +- 2.1 ms | 100.6% l-d-r | x | x | 190.2 ms +- 4.8 ms | 191.6 ms +- 2.0 ms | 100.7% rebase -r . --keep -d .^^: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 5.655 s +- 0.029 s | 5.640 s +- 0.036 s | 99.7% m-u | | x | 5.813 s +- 0.038 s | 5.773 s +- 0.028 s | 99.3% m-u | x | | 5.593 s +- 0.043 s | 5.589 s +- 0.028 s | 99.9% m-u | x | x | 648.2 ms +- 19.2 ms | 637.3 ms +- 27.7 ms | 98.3% l-d-r | | | 673.3 ms +- 8.0 ms | 673.2 ms +- 6.8 ms | 100.0% l-d-r | | x | 6.583 s +- 0.030 s | 5.721 s +- 0.028 s | 86.9% <-- l-d-r | x | | 277.8 ms +- 6.7 ms | 276.0 ms +- 2.7 ms | 99.4% l-d-r | x | x | 1.692 s +- 0.013 s | 720.9 ms +- 13.3 ms | 42.6% <-- status --change . --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 220.9 ms +- 1.6 ms | 219.9 ms +- 2.2 ms | 99.5% m-u | | x | 109.2 ms +- 1.0 ms | 109.4 ms +- 0.8 ms | 100.2% m-u | x | | 222.6 ms +- 1.7 ms | 221.4 ms +- 2.1 ms | 99.5% m-u | x | x | 113.4 ms +- 0.5 ms | 113.1 ms +- 1.1 ms | 99.7% l-d-r | | | 82.1 ms +- 1.7 ms | 82.1 ms +- 1.2 ms | 100.0% l-d-r | | x | 199.8 ms +- 4.0 ms | 200.7 ms +- 3.6 ms | 100.5% l-d-r | x | | 85.4 ms +- 1.5 ms | 85.2 ms +- 0.3 ms | 99.8% l-d-r | x | x | 202.6 ms +- 4.4 ms | 208.0 ms +- 4.0 ms | 102.7% status --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.941 s +- 0.014 s | 1.930 s +- 0.009 s | 99.4% m-u | | x | 1.924 s +- 0.007 s | 1.950 s +- 0.010 s | 101.4% m-u | x | | 1.959 s +- 0.085 s | 1.926 s +- 0.009 s | 98.3% m-u | x | x | 96.2 ms +- 1.0 ms | 96.4 ms +- 0.7 ms | 100.2% l-d-r | | | 604.4 ms +- 10.6 ms | 602.6 ms +- 7.1 ms | 99.7% l-d-r | | x | 605.7 ms +- 4.1 ms | 607.4 ms +- 6.1 ms | 100.3% l-d-r | x | | 182.4 ms +- 1.2 ms | 183.4 ms +- 1.2 ms | 100.5% l-d-r | x | x | 150.8 ms +- 2.0 ms | 150.6 ms +- 1.0 ms | 99.9% update $rev^; ~/src/hg/hg{hg}/hg update $rev: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 3.185 s +- 0.027 s | 3.181 s +- 0.017 s | 99.9% m-u | | x | 3.028 s +- 0.021 s | 2.954 s +- 0.010 s | 97.6% m-u | x | | 3.168 s +- 0.010 s | 3.175 s +- 0.023 s | 100.2% m-u | x | x | 317.5 ms +- 3.5 ms | 313.2 ms +- 2.9 ms | 98.6% l-d-r | | | 456.2 ms +- 10.6 ms | 454.4 ms +- 5.8 ms | 99.6% l-d-r | | x | 9.236 s +- 0.063 s | 757.9 ms +- 9.2 ms | 8.2% <-- l-d-r | x | | 257.6 ms +- 2.3 ms | 261.2 ms +- 1.7 ms | 101.4% l-d-r | x | x | 1.614 s +- 0.013 s | 478.0 ms +- 14.3 ms | 29.6% <-- Differential Revision: https://phab.mercurial-scm.org/D4875

treemanifests: store whether a lazydirs entry needs copied after materializing Due to the way that things like manifestlog.get caches its values, without making a copy (if necessary) after calling readsubtree(), we might end up adjusting the state of the same object on different contexts, breaking things like dirty state tracking (and probably other things). Differential Revision: https://phab.mercurial-scm.org/D4874

treemanifests: extract _loaddifflazy from _diff, use in _filesnotin Differential Revision: https://phab.mercurial-scm.org/D4873

identify: show remote bookmarks in `hg id url -Tjson -B` I didn't display bookmarks when `default and not ui.quiet`: it seems strange for templates to depend on --id or -q, and it would take more code for `hg id url -T {node}` to not request remote bookmarks. An alternative I thought of was providing lazy data to the formatter, `fm.data(bookmarks=lambda: fm.formatlist(getbms(), name='bookmark'))`. The plainformatter would naturally not compute it, the templateformatter would compute only what it needs, and the other ones would compute everything, but that's not supported (or I don't see how), so I abandoned this idea. Differential Revision: https://phab.mercurial-scm.org/D4872

showstack: also handle SIGALRM This is looking *very* handy when debugging mysterious hangs in a test: you can wrap a hanging invocation in `perl -e 'alarm shift @ARGV; exec @ARGV' 1` for example, a hanging `hg pull` becomes `perl -e 'alarm shift @ARGV; exec @ARGV' 1 hg pull` where the `1` is the timeout in seconds before the process will be hit with SIGALRM. After making that edit to the test file, you can then use --extra-config-opt on run-tests.py to globaly enable showstack during the test run, so you'll get full stack traces as you force your hg to exit. I wonder (but only a little, not enough to take action just yet) if we should wire up some scaffolding in run-tests itself to automatically wrap all commands in alarm(3) somehow to avoid hangs in the future? Differential Revision: https://phab.mercurial-scm.org/D4870

exchangev2: add progress bar around manifest scanning This can take a long time on large repositories. Let's add a progress bar so we don't have long periods where it isn't obvious what is going on. Differential Revision: https://phab.mercurial-scm.org/D4859

httppeer: report http statistics Now that keepalive.py records HTTP request count and the number of bytes sent and received as part of performing those requests, we can easily print a report on the activity when closing a peer instance! Exact byte counts are globbed in tests because they are influenced by non-deterministic things, such as hostnames and port numbers. Plus, the exact byte count isn't too important anyway. I feel obliged to note that printing the byte count could have security implications. e.g. if sending a password via HTTP basic auth, the length of that password will influence the byte count and the reporting of the byte count could be a side-channel leak of the password length. I /think/ this is beyond our threshold for concern. But if we think it poses a problem, we can teach the byte count logging code to e.g. ignore sensitive HTTP request headers. We could also consider not reporting the byte count of request headers altogether. But since the wire protocol uses HTTP headers for sending command arguments, it is kind of important to report their size. Differential Revision: https://phab.mercurial-scm.org/D4858

keepalive: track number of bytes received from an HTTP response We also bubble the byte count up to the HTTPConnection instance and its parent opener at read time. Unlike sending, there isn't a clear "end of response" signal we can intercept to defer updating the accounting. Differential Revision: https://phab.mercurial-scm.org/D4857

keepalive: track request count and bytes sent I want wire protocol interactions to report the number of requests made and bytes transferred. This commit teaches the very low-level custom HTTPConnection class to track the number of bytes sent to the socket. This may vary from the number of bytes that go on the wire due to e.g. TLS. That's OK. KeepAliveHandler is taught to track the total number of requests and total number of bytes sent across all requests. Differential Revision: https://phab.mercurial-scm.org/D4856

url: have httpsconnection inherit from our custom HTTPConnection This will ensure that any customizations we perform to HTTPConnection will be available to httpsconnection. Differential Revision: https://phab.mercurial-scm.org/D4855

cborutil: change buffering strategy Profiling revealed that we were spending a lot of time on the line that was concatenating the old buffer with the incoming data when attempting to decode long byte strings, such as manifest revisions. Essentially, we were feeding N chunks of size len(X) << len(Y) into decode() and continuously allocating a new, larger buffer to hold the undecoded input. This created substantial memory churn and slowed down execution. Changing the code to aggregate pending chunks in a list until we have enough data to fully decode the next atom makes things much more efficient. I don't have exact data, but I recall the old code spending >1s on manifest fulltexts from the mozilla-unified repo. The new code doesn't significantly appear in profile output. Differential Revision: https://phab.mercurial-scm.org/D4854

cleanup: some Yoda conditions, this patch removes It seems the factor 20 is less than the frequency of " < \d" compared to " \d > ". Differential Revision: https://phab.mercurial-scm.org/D4862

streamclone: don't support stream clone unless repo feature present This change means custom repository types must opt in to enabling stream clone. This seems reasonable, as stream clones are a very low-level feature that has historically assumed the use of revlogs and the layout of .hg/ that they entail. Differential Revision: https://phab.mercurial-scm.org/D4853

localrepo: add repository feature when repo can be stream cloned Right now, the wire protocol server assumes all repository objects can be stream cloned (unless the stream clone feature is disabled via config option). But not all storage backends or repository objects may support stream clone. This commit defines a repository feature denoting whether stream clone is supported. The feature is defined for revlog-based repositories, which should currently be "all repositories." Differential Revision: https://phab.mercurial-scm.org/D4852

wireprotov2: client support for following content redirects And with the server actually sending content redirects, it is finally time to implement client support for following them! When a redirect response is seen, we wait until all data for that request has been received (it should be nearly immediate since no data is expected to follow the redirect message). Then we use a URL opener to make a request. We stuff that response into the client handler and construct a new response object to track it. When readdata() is called for servicing requests, we attempt to read data from the first redirected response. During data reading, data is processed similarly to as if it came from a frame payload. The existing test for the functionality demonstrates the client transparently following the redirect and obtaining the command response data from an alternate URL! There is still plenty of work to do here, including shoring up testing. I'm not convinced things will work in the presence of multiple redirect responses. And we don't yet implement support for integrity verification or configuring server certificates to validate the connection. But it's a start. And it should enable us to start experimenting with "real" caches. Differential Revision: https://phab.mercurial-scm.org/D4778

wireprotov2: server support for sending content redirects A "content redirect" can be sent in place of inline response content. In terms of code, we model a content redirect as a special type of response object holding the attributes describing that redirect. Sending a content redirect thus becomes as simple as the object emission layer sending an instance of that type. A cacher using externally-addressable content storage could replace the outgoing object stream with an object advertising its location. The bulk of the code in this commit is teaching the output layer which handles the object stream to recognize alternate location objects. The rules are that if an alternate location object is present, it must be the first and only object in the object stream. Otherwise the server emits an error. Differential Revision: https://phab.mercurial-scm.org/D4777

wireprotov2: client support for advertising redirect targets With the server now able to emit a redirect target descriptor, we can start to teach the client to recognize it. This commit implements support for filtering the advertised redirect targets against supported features and for advertising compatible redirect targets as part of command requests. It also adds the minimal boilerplate required to fail when a content redirect is seen. The server doesn't yet do anything with the advertised redirect targets. And the client can't yet follow redirects if it did. But at least we're putting bytes on the wire. Differential Revision: https://phab.mercurial-scm.org/D4776

wireprotov2: advertise redirect targets in capabilities This is pretty straightforward. Redirect targets will require an extension to support. So we've added a function that can be wrapped to define redirect targets. To test this, we teach our simple cache test extension to read redirect targets from a file. It's a bit hacky. But it gets the job done. Differential Revision: https://phab.mercurial-scm.org/D4775

wireprotov2: define semantics for content redirects When I implemented the clonebundles feature and deployed it on hg.mozilla.org using Amazon S3 as a content server, server-side CPU and bandwidth usage dropped off a cliff and a ton of server scaling headaches went away pretty much the instant clients with support for clonebundles were rolled out to Firefox CI. An obvious takeaway from that experience was that offloading server load to scalable file servers - potentially backed by a CDN - is a really good idea. Another takeaway was that Mercurial's wire protocol wasn't in a good position to support data offload generally. In wire protocol version 1, there isn't a mechanism in the protocol to say "grab the data from over here instead." For HTTP, we could teach the client to follow HTTP redirects. Or we could invent a media type that encoded redirects inline. But for SSH, we were pretty much out of luck because that protocol wasn't very flexible. Wire protocol version 2 offers the opportunity to do something better. The recent generic server-side content caching layer in the wire protocol version 2 server demonstrated that it is possible to have drop-in caching of responses to command requests. This by itself adds tons of value and already makes the built-in server much more scalable. But I don't want to stop there. The existing server-side caching implementation has a big weakness: it requires the server to send data to the client. This means that the Mercurial server is potentially sending gigabytes of data to thousands of clients. This is problematic because compared to scaling static file servers, scaling dynamic servers is *hard*. A solution to this is to "offload" serving of content to something that isn't the Mercurial server. By offloading content serving, you turn the Mercurial server from a centralized monolithic service to a distributed mostly-indexing service. Assuming high rates of content offload, this should drastically reduce the total work performed by the Mercurial server, both in terms of CPU and data transfer. This will make Mercurial servers vastly easier to scale. This commit defines the semantics for "content redirects" in wire protocol version 2. Essentially: * Servers advertise the set of locations a response could be served from. * When making requests, clients advertise the set of locations they are willing to fetch content from. * Servers can then replace the inline response with one that says "get the response from over here instead." This feature - when fully implemented - will allow extending the server-side caching layer to facilitate such things as integrating your server-side cache with a scalable blob store (such as S3 or a CDN) and offloading most data transfer to that external service. This feature could also be leveraged for load balancing. e.g. requests could come into a central server and then get redirected to an available mirror depending on server availability or locality. There's tons of potential :) Differential Revision: https://phab.mercurial-scm.org/D4774

wireprotov2: support response caching One of the things I've learned from managing VCS servers over the years is that they are hard to scale. It is well known that some companies have very beefy (read: very expensive) servers to power their VCS needs. It is also known that specialized servers for various VCS exist in order to facilitate scaling servers. (Mercurial is in this boat.) One of the aspects that make a VCS server hard to scale is the high CPU load incurred by constant client clone/pull operations. To alleviate the scaling pain associated with data retrieval operations, I want to integrate caching into the Mercurial wire protocol server as robustly as possible such that servers can aggressively cache responses and defer as much server load as possible. This commit represents the initial implementation of a general caching layer in wire protocol version 2. We define a new interface and behavior for a wire protocol cacher in repository.py. (This is probably where a reviewer should look first to understand what is going on.) The bulk of the added code is in wireprotov2server.py, where we define how a command can opt in to being cached and integrate caching into command dispatching. From a very high-level: * A command can declare itself as cacheable by providing a callable that can be used to derive a cache key. * At dispatch time, if a command is cacheable, we attempt to construct a cacher and use it for serving the request and/or caching the request. * The dispatch layer handles the bulk of the business logic for caching, making cachers mostly "dumb content stores." * The mechanism for invalidating cached entries (one of the harder parts about caching in general) is by varying the cache key when state changes. As such, cachers don't need to be concerned with cache invalidation. Initially, we've hooked up support for caching "manifestdata" and "filedata" commands. These are the simplest to cache, as they should be immutable over time. Caching of commands related to changeset data is a bit harder (because cache validation is impacted by changes to bookmarks, phases, etc). This will be implemented later. (Strictly speaking, censoring a file should invalidate caches. I've added an inline TODO to track this edge case.) To prove it works, this commit implements a test-only extension providing in-memory caching backed by an lrucachedict. A new test showing this extension behaving properly is added. FWIW, the cacher is ~50 lines of code, demonstrating the relative ease with which a cache can be added to a server. While the test cacher is not suitable for production workloads, just for kicks I performed a clone of just the changeset and manifest data for the mozilla-unified repository. With a fully warmed cache (of just the manifest data since changeset data is not cached), server-side CPU usage dropped from ~73s to ~28s. That's pretty significant and demonstrates the potential that response caching has on server scalability! Differential Revision: https://phab.mercurial-scm.org/D4773

wireprotov2: define type to represent pre-encoded object An upcoming commit will introduce a caching layer to command serving. This will require the ability to cache pre-encoded data. This commit introduces a type to represent pre-encoded data and teaches the output layer to not CBOR encode an instance of that type. Differential Revision: https://phab.mercurial-scm.org/D4772

wireprotov2: change name and behavior of readframe() In the near future, we will want to support performing I/O from other sources. Let's rename readframe() to readdata() and tweak its logic to support future growth. Differential Revision: https://phab.mercurial-scm.org/D4771

url: move _wraphttpresponse() from httpeer This is a generally useful function. Having it on the url module will make it more accessible outside of the HTTP peers. Differential Revision: https://phab.mercurial-scm.org/D4770

debugcommands: print all CBOR objects application/mercurial-cbor may contain multiple objects. Let's print all of them. Differential Revision: https://phab.mercurial-scm.org/D4769

help: document about "export" template keywords

help: document about "config" template keywords

help: document about "cat" template keywords

help: document about "branches" template keywords

help: document about "bookmarks" template keywords

help: document about "annotate" template keywords

storageutil: pass nodes into emitrevisions() The main emitrevisions() uses nodes. So it makes sense to use nodes for the helper API. Not bothering with API since this function was introduced a few commits ago. Differential Revision: https://phab.mercurial-scm.org/D4805

storageutil: make all callables optional Not all storage backends may implement these callables. That's part of the reason these methods aren't exposed on the storage interface. Differential Revision: https://phab.mercurial-scm.org/D4804

storageutil: extract most of emitrevisions() to standalone function As part of implementing a storage backend, I found myself copying most of revlog.emitrevisions(). This code is highly nuanced and it bothered me greatly to be copying such low-level code. This commit extracts the bulk of revlog.emitrevisions() into a new standalone function. In order to make the function generally usable, all "self" function calls that aren't exposed on the ifilestorage interface are passed in via callable arguments. No meaningful behavior should have changed as part of the port. Upcoming commits will tweak behavior to make the code more generically usable. Differential Revision: https://phab.mercurial-scm.org/D4803

storageutil: invert logic of file data comparison IMO things make more sense when the function is explicitly a test for file data equivalence. Not bothering with API since the function was introduced by the previous commit. Differential Revision: https://phab.mercurial-scm.org/D4802

storageutil: extract filelog.cmp() to a standalone function As part of implementing an alternate storage backend, I found myself reimplementing this code. With a little massaging, we can extract filelog.cmp() to a standalone function. As part of this, the call to revlog.cmp() was inlined (it is just a 2-line function). I also tweaked some variable names to improve readability. I'll further tweak names in a subsequent commit. Differential Revision: https://phab.mercurial-scm.org/D4801

storageutil: extract copy metadata retrieval out of filelog As part of implementing an alternate storage backend, I found myself reinventing this wheel. Let's create a utility function for doing the work. Differential Revision: https://phab.mercurial-scm.org/D4800

storageutil: extract functionality for resolving strip revisions I found myself having to copy this method as part of implementing a new storage backend. With a little tweaking, we can extract it to a standalone function so it can be reused easily. We'll likely want to implement a better method for removing revisions on the storage interface someday. But until then, let's use what we have. Differential Revision: https://phab.mercurial-scm.org/D4799

storageutil: consistently raise LookupError (API) The interface docs say this is supposed to raise LookupError on failure. But for invalid revision number input, it could raise IndexError because ifileindex.node() is documented to raise IndexError. lookup() for files isn't used that much (pretty much just in basefilectx in core AFAICT). And callers should already be catching LookupError. So I don't anticipate that much fallout from this change. Differential Revision: https://phab.mercurial-scm.org/D4798

storageutil: implement file identifier resolution method (BC) revlog.lookup() has a number of advanced features, including partial node matching. These advanced features aren't needed for file id lookup because file identifiers are almost always from internal sources. (An exception to this is hgweb, which appears to have some code paths that attempt to resolve a user-supplied string to a node.) This commit implements a new function for resolving file identifiers to nodes. It behaves like revlog.lookup() but without the advanced features. Tests reveal behavior changes: * Partial hex nodes are no longer resolved to nodes. * "-1" now returns nullid instead of raising LookupError. * "0" on an empty store now raises LookupError instead of returning nullid. I'm not sure why "-1" wasn't recognized before. But it seems reasonable to accept it as a value since integer -1 resolves to nullid. These changes all seem reasonable to me. And with the exception of partial hex node matching, we may want to consider changing revlog.lookup() as well. Differential Revision: https://phab.mercurial-scm.org/D4797

testing: add more testing for ifileindex.lookup() The tests demonstrate some... questionable behavior of revlog.lookup(). Differential Revision: https://phab.mercurial-scm.org/D4796

dagop: extract DAG local heads functionality from revlog As part of implementing an alternate storage backend, I found myself having to reimplement this function. The DAG traversal logic is generic and can be factored out into a standalone function. We could probably combine this with headrevs(). But I'll leave that for another time. Differential Revision: https://phab.mercurial-scm.org/D4795

dagop: extract descendants() from revlog module This method needs to be implemented in other storage backends and is generic if we parameterize the functions for retrieving revision numbers and parent revision numbers. Let's extract it to dagop so backends don't need to reinvent the wheel. I'm not too worried about performance overhead of the extra function call because I don't expect anyone to be calling descendants() in a tight loop. Differential Revision: https://phab.mercurial-scm.org/D4794

filelog: remove checkhash() (API) It is unused. While a caller may want to ask the store to validate the hash of some provided text, since there are no in-core consumers of this method, let's drop it for now. Differential Revision: https://phab.mercurial-scm.org/D4793

filelog: remove revdiff() (API) This proxy method is no longer used. While it might be useful to query a storage backend for the delta between any 2 revisions because the store could have a delta cached and could compute it more efficiently than the caller calling revision() twice in order to compute a delta, since nothing in core is using this API now, I feel comfortable nuking it. Differential Revision: https://phab.mercurial-scm.org/D4792

localrepo: define storage backend in creation options (API) We add an experimental config option to define the storage backend for new repositories. By default, it uses "revlogv1," which maps to the current and only modern supported repository format. We add a "backend" creation option to control which backend to use. It defaults to using the value from the config option. newreporequirements() will now barf if it sees a "backend" value that isn't "revlogv1." This forces extensions to monkeypatch the function to handle requirements derivation for custom backends. In order for this to "just work," we factored out obtaining the default creation options into its own function and made callers of newreporequirements() responsible for passing in valid data. Without this, direct callers of newreporequirements() wouldn't get the proper results. Differential Revision: https://phab.mercurial-scm.org/D4791

wireprotov2: derive "required" from presence of default value If we define a default value for all optional arguments, we no longer need to explicitly declare whether the argument is required. Instead, we can derive it from the presence of a default value callable. Differential Revision: https://phab.mercurial-scm.org/D4790

localrepo: capture repo interface factory functions as lambas Previously, the list would hold a reference to the original function and attempting to wrap the module function via extension wouldn't result in the wrapped function being called, completely undermining the expected behavior. Differential Revision: https://phab.mercurial-scm.org/D4821

extensions: new closehead module for closing arbitrary heads ``hg close-head`` allows closing arbitrary heads. It is equivalent to checking out the given revisions and commit an empty change with ``hg commit --close-branch``. Differential Revision: https://phab.mercurial-scm.org/D3557

cext: use modern buffer protocol in mpatch_flist() Differential Revision: https://phab.mercurial-scm.org/D4841

cext: use modern buffer protocol in patches() PyObject_AsCharBuffer() is part of the "Old Buffer Protocol," which has been deprecated for years. Let's port away from it. PyBuffer_GetBuffer() must be paired with PyBuffer_Release(), hence the added "goto cleanup" in a failure case. We don't bump the extension version because the API has not changed. Differential Revision: https://phab.mercurial-scm.org/D4840

identify: when using -T, avoid unnecessary remote bookmarks query Differential Revision: https://phab.mercurial-scm.org/D4839

identify: only query remote bookmarks if needed Instead of all the time when operating on a remote repo. This perf regression was introduced in 15a79ac823e8, in 4.3. This datahint method returns nothing for -Tjson, -Tpickle, -Tdebug --config ui.formatdebug=true and --config ui.formatjson, so the bookmarks won't show up. I don't know what these formatters are for. plainformatter and templateformatter work properly, and the few other uses of datahint should have the same kind of problem. There is further weirdness where "--template '{node}'" is not enough to avoid querying the bookmarks, you also need to pass --id or -q. Differential Revision: https://phab.mercurial-scm.org/D4819

py3: whitelist another passing tests caught by buildbot I this indygreg recent patches made this test pass on Python 3. So thanks to him! Differential Revision: https://phab.mercurial-scm.org/D4848

manifest: remove an unused variable caught by pyflakes Uses of todel were removed in D4843 (19103e68a698). This patch will make buildbots green again. Differential Revision: https://phab.mercurial-scm.org/D4847

setup: ignore message about disabling 3rd party extensions because of version I started getting into a bind recently when switching between py2 and py3 because switching requires a `make clean`, which kills __version__.py. But then when running `make local`, it picks up the local hg.exe (MSYS seems to prefix $PATH with '.'), which doesn't know its version. That causes it to emit a warning about needing at least 4.3 to load evolve, which caused setup.py to fail saying there is no working hg executable to figure out the version. If we can ignore general extension import failures, we should be able to ignore this too.

narrow: avoid overwriting a variable I don't like to overwrite variables, especially not with a different type (because it makes it harder to explain what the variable is for, and, therefore, to name it well). Differential Revision: https://phab.mercurial-scm.org/D4846

treemanifests: remove _loadalllazy in _diff() The benchmarks below use a similar setup as in ee7ee0c516ca and my other recent commits. Yes, in some cases this runs in literally 5% of the time it previously took. before = a6f8ab53 diff --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.301 s +- 0.004 s | 1.309 s +- 0.012 s | 100.6% m-u | | x | 1.303 s +- 0.009 s | 1.302 s +- 0.006 s | 99.9% m-u | x | | 1.308 s +- 0.006 s | 1.309 s +- 0.007 s | 100.1% m-u | x | x | 85.7 ms +- 0.6 ms | 86.0 ms +- 0.3 ms | 100.4% l-d-r | | | 197.5 ms +- 0.7 ms | 197.8 ms +- 2.2 ms | 100.2% l-d-r | | x | 199.4 ms +- 0.6 ms | 199.3 ms +- 0.9 ms | 99.9% l-d-r | x | | 86.1 ms +- 0.5 ms | 85.8 ms +- 0.9 ms | 99.7% l-d-r | x | x | 64.4 ms +- 0.4 ms | 64.4 ms +- 0.3 ms | 100.0% diff -c . --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 236.7 ms +- 1.1 ms | 236.5 ms +- 1.3 ms | 99.9% m-u | | x | 158.7 ms +- 1.0 ms | 128.0 ms +- 1.0 ms | 80.7% <-- m-u | x | | 239.7 ms +- 1.8 ms | 238.1 ms +- 1.5 ms | 99.3% m-u | x | x | 132.4 ms +- 0.9 ms | 132.3 ms +- 0.6 ms | 99.9% l-d-r | | | 81.8 ms +- 0.4 ms | 81.8 ms +- 0.3 ms | 100.0% l-d-r | | x | 3.894 s +- 0.017 s | 193.6 ms +- 0.4 ms | 5.0% <-- l-d-r | x | | 106.9 ms +- 0.4 ms | 106.6 ms +- 0.3 ms | 99.7% l-d-r | x | x | 182.7 ms +- 0.8 ms | 183.3 ms +- 0.9 ms | 100.3% rebase -r . --keep -d .^^: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 5.615 s +- 0.109 s | 5.562 s +- 0.015 s | 99.1% m-u | | x | 5.701 s +- 0.027 s | 5.715 s +- 0.023 s | 100.2% m-u | x | | 5.572 s +- 0.128 s | 5.613 s +- 0.182 s | 100.7% m-u | x | x | 633.3 ms +- 28.7 ms | 636.2 ms +- 13.8 ms | 100.5% l-d-r | | | 666.4 ms +- 17.0 ms | 658.5 ms +- 9.3 ms | 98.8% l-d-r | | x | 6.520 s +- 0.070 s | 6.505 s +- 0.026 s | 99.8% l-d-r | x | | 279.0 ms +- 13.0 ms | 276.5 ms +- 4.7 ms | 99.1% l-d-r | x | x | 1.636 s +- 0.058 s | 1.657 s +- 0.014 s | 101.3% status --change . --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 218.6 ms +- 1.4 ms | 217.9 ms +- 1.5 ms | 99.7% m-u | | x | 138.5 ms +- 0.4 ms | 108.4 ms +- 0.2 ms | 78.3% <-- m-u | x | | 220.1 ms +- 1.3 ms | 219.7 ms +- 1.5 ms | 99.8% m-u | x | x | 113.2 ms +- 0.4 ms | 112.4 ms +- 0.8 ms | 99.3% l-d-r | | | 80.2 ms +- 0.3 ms | 80.6 ms +- 0.6 ms | 100.5% l-d-r | | x | 3.899 s +- 0.020 s | 194.8 ms +- 4.0 ms | 5.0% <-- l-d-r | x | | 83.4 ms +- 0.8 ms | 83.2 ms +- 0.2 ms | 99.8% l-d-r | x | x | 732.2 ms +- 4.3 ms | 194.9 ms +- 1.0 ms | 26.6% <-- status --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.917 s +- 0.005 s | 1.914 s +- 0.004 s | 99.8% m-u | | x | 1.909 s +- 0.012 s | 1.934 s +- 0.004 s | 101.3% m-u | x | | 1.915 s +- 0.005 s | 1.904 s +- 0.004 s | 99.4% m-u | x | x | 94.8 ms +- 0.3 ms | 94.7 ms +- 0.2 ms | 99.9% l-d-r | | | 593.9 ms +- 1.2 ms | 594.6 ms +- 9.4 ms | 100.1% l-d-r | | x | 595.2 ms +- 3.8 ms | 597.2 ms +- 2.6 ms | 100.3% l-d-r | x | | 182.5 ms +- 1.6 ms | 182.1 ms +- 0.6 ms | 99.8% l-d-r | x | x | 149.6 ms +- 0.9 ms | 149.1 ms +- 0.8 ms | 99.7% update $rev^; ~/src/hg/hg{hg}/hg update $rev: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 3.121 s +- 0.007 s | 3.129 s +- 0.012 s | 100.3% m-u | | x | 2.972 s +- 0.011 s | 2.981 s +- 0.012 s | 100.3% m-u | x | | 3.144 s +- 0.014 s | 3.141 s +- 0.011 s | 99.9% m-u | x | x | 312.2 ms +- 2.4 ms | 312.3 ms +- 2.1 ms | 100.0% l-d-r | | | 444.4 ms +- 4.3 ms | 446.9 ms +- 5.3 ms | 100.6% l-d-r | | x | 9.159 s +- 0.069 s | 9.182 s +- 0.040 s | 100.3% l-d-r | x | | 254.6 ms +- 1.6 ms | 255.2 ms +- 1.6 ms | 100.2% l-d-r | x | x | 1.525 s +- 0.007 s | 1.577 s +- 0.007 s | 103.4% <--? Differential Revision: https://phab.mercurial-scm.org/D4845

treemanifests: skip extraneous check for item before calling _loadlazy Differential Revision: https://phab.mercurial-scm.org/D4844

treemanifests: make _loadchildrensetlazy just call _loadlazy This was suggested on the original review for _loadchildrensetlazy (D4370), it's taken me a while to get to it though. Differential Revision: https://phab.mercurial-scm.org/D4843

treemanifests: make _loadlazy tolerate item not on _lazydirs I'd like to clean up a few cases where we check for an item in _lazydirs before calling _loadlazy - this will remove an extraneous dict lookup and make it slightly more versatile. Differential Revision: https://phab.mercurial-scm.org/D4842

debugcommands: add a debugindexstats command Someone went through the trouble of recording these stats, so let's make them accessible. Differential Revision: https://phab.mercurial-scm.org/D4832

rust-chg: add main program It can at least connect to a *running* command server, and execute Mercurial command. No signal handling nor daemon management is implemented yet.

rust-chg: add interface to chdir the server

rust-chg: add Client extensions to run cHg-specific requests This just provides a nicer way to issue command-server requests.

rust-chg: port basic socket path handling from cHg of C This is basically modeled after setcmdserveropts() of chg.c.

rust-chg: add state machine to handle "runcommand" request with cHg extension This is modeled after tokio-hglib's RunCommand state to support the "S" channel message.

rust-chg: add callback to handle pager and shell command requests This could be inlined into the ChgRunCommand state to be introduced by the next patch, but it seemed good to separate any user interactions from the IPC code.

rust-chg: add low-level function to set pager fd blocking This is necessary because the server expects stdout/stderr to be blocking, whereas we'll use async library to spawn pager, which makes pipes unblocking.

rust-chg: add future that handles "attachio" request This is the sequence to send client-side stdio and pager stdin to the server.

rust-chg: add parser for request messages sent to "S" channel The data structure is documented at chgserver.py:channeledsystem().

rust-chg: add wrapper around C function

rust-chg: add function to send fds via domain socket As a beginning, I wrote some C. It's extracted from attachio() of contrib/chg/hgclient.c. Maybe it could be rewritten in Rust by using the libc (and/or nix) crates, but doing that wouldn't be trivial as the code depends on CMSG_*() macros. IMO, using C is better here.

rust-chg: update dependencies

rust-chg: add project skeleton This directory will host the reimplementation of cHg in Rust. It will use Tokio [1], and tokio-hglib [2] which I wrote for an oxidized CHg, no idea if there's such carbon bonding in nature btw. [1]: https://tokio.rs/ [2]: https://bitbucket.org/yuja/tokio-hglib/ The reasoning for depending on Tokio is that it will allow us to handle Unix signals in a safer way. Well, I believed that until I found a weird function, handlestopsignal(), in cHg codebase. It resends the same signal to the same process by temporarily masking the handler, which can't be inherently async. So the signal handlers will stay in C, which means there isn't actually much reason to write async code right now, other than I've already done most of the async stuff, and slightly easier pager handling. The reasoning for the rewrite is that it will eventually be possible to port server-side config validation back to the client side, which will reduce the complexity of the current daemon management. It will also encourage us to write frontend library (e.g. command line and config parsers) in Rust. The license is GPL2+ because it's likely to include derived work from the cHg of C. The rust/chg crate is excluded from the root workspace as it's unclear how the whole rust packages should be laid out. That can be revisited later.

narrow: move copies overrides to core The copies overrides seems to have been a little complicated just by not being in core. When moved to core, it becomes trivial (at least I think these overrides have the same effect). Differential Revision: https://phab.mercurial-scm.org/D4825

narrow: pass old includes and excludes to _widen() In future patches we will need to pass them in the widen wireprotocol command which we are building. Differential Revision: https://phab.mercurial-scm.org/D4812

narrow: check for servers' narrow support before doing anything (BC) Recently we introduced narrow capabilities for the server. So we can check whether a server has narrow clone support or not before doing anything. This is BC because new clients won't be able to extend from old narrow-enabled servers. I *think* narrow is not used much (maybe just inside Google), also it's experimental so I think we can change this. We will need to this someday anyway. The "doesn't" in error is changed to "does not" because I think that's we do in core. I also changed one more instance of the error message to use 'does not' for consistency. Differential Revision: https://phab.mercurial-scm.org/D4789

narrow: don't do the dirstate dance if ellipses is not enabled I believe we set dirstate parents to nullid before widening pull because in ellipses cases, the parent might be stripped off with a new changeset. However the second ds.setparents() call invalidate my assumption. I am not sure why we do this. So here is a patch. This patch also adds tests showing we break nothing in non-ellipses cases. Differential Revision: https://phab.mercurial-scm.org/D4788

narrow: pass 'narrow_widen' as source while generating changegroup Extensions inspect the `source` parameter and add some custom logic on the basis of value of that parameter. The parameter is also passed to hooks. So let's pass the value as 'narrow_widen' to differentiate widening from pull. Differential Revision: https://phab.mercurial-scm.org/D4787

narrow: factor out logic to create cg while widening into separate fn This patch takes out the logic which generates a changegroup for widening a narrow clone when ellipses are disabled. This is done because future patches will introduce a narrow_widen() wireprotocol command which will send a bundle2 with changegroup and will use this function. The new function for now returns just the changegroup for compatibility with existing code, but in future patches, when we establish a wireprotocol command and call this function from there, this will return the required bundle2. Differential Revision: https://phab.mercurial-scm.org/D4786

narrow: avoid looking up dirstate again when editing dirstate The narrow extension overrides the dirstate editing functions to restrict paths outside the narrowspec. These overrides access the dirstate from repo.dirstate instead of using the "self" reference passed to the overridden functions. I don't see a reason for this and it caused me problems with a later patch (it caused infinite recursion when I modified localrepo.dirstate()), so let's change it. Differential Revision: https://phab.mercurial-scm.org/D4829

repo: move unfiltered-repo optimization to workingctx localrepo.__getitem__ special-cased lookup of the working copy parent to avoid looking up obsmarkers. I think the reason for that code (which I once wrote myself) was to make `hg commit` not load obsmarkers, which it would otherwise do via ctx.p1() in localrepo.commitctx(). That had the somewhat unfortunate consequence of making lookup of an unrelated binary nodeid load the dirstate. Now that changectx's constructor is dumb, we can let workingctx._parents() have the opmtimization instead. This affects two tests, because they no longer end up loading the dirstate and their "warning: ignoring unknown working parent ..." messages therefore go way. Differential Revision: https://phab.mercurial-scm.org/D4828

context: move logic from changectx.__init__ to localrepo.__getitem__ (API) My motivation for this change was to make repo[node] not load the dirstate (more about that in the next patch), but I think it makes more sense this way too. For example, raising RepoLookupError seems to belong better in the repo lookup function (i.e. localrepo.__getitem__). This makes the changectx constructor very simple -- it just assigns the given repo, revnum, and nodeid to properties. Differential Revision: https://phab.mercurial-scm.org/D4827

context: reduce dependence of changectx constructor I want to change the constructor's signature and letting all creation of changectx instances go through the repo simplifies that. Differential Revision: https://phab.mercurial-scm.org/D4826

py3: use system strings for HTTP response header comparison res.headers is holding system strings. Attempting to compare against bytes always fails and we fail to print decoded CBOR responses on Python 3. After this change, various test-wireproto* tests are now properly printing CBOR response objects. Differential Revision: https://phab.mercurial-scm.org/D4835

py3: convert HTTP request headers to str The low-level request object ideally takes system strings for HTTP request headers and values. If we send in bytes, it works. But a duplicate header check fails and various tests emit duplicate user-agent headers. Differential Revision: https://phab.mercurial-scm.org/D4834

py3: use system strings in HTTP server code Previously the source transformer was converting some string literals to bytes and we were comparing a system native string to bytes and this was leading to sending the wrong HTTP response headers on Python 3. After this change, we now properly send Transfer-Encoding and Connection response headers on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4833

py3: byteify test-storage.py The test now passed on Python 3. # skip-blame just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4836

narrow: remove narrowpatch instead of narrowcopies 84092edd5c88 removed narrowpatch.py and wanted to drop narrowpatch from __init__.py but mistakenly removed narrowcopies. This will make buildbots green again. Differential Revision: https://phab.mercurial-scm.org/D4837

py3: suppress the output from .write() calls in more tests I missed these in 803b7569c9ea because I forgot about `...` lines.

py3: byteify test-lfs.t