view contrib/all-revsets.txt @ 40021:c537144fdbef

wireprotov2: support response caching

One of the things I've learned from managing VCS servers over the
years is that they are hard to scale. It is well known that some
companies have very beefy (read: very expensive) servers to power
their VCS needs. It is also known that specialized servers for
various VCS exist in order to facilitate scaling servers. (Mercurial
is in this boat.)

One of the aspects that make a VCS server hard to scale is the
high CPU load incurred by constant client clone/pull operations.

To alleviate the scaling pain associated with data retrieval
operations, I want to integrate caching into the Mercurial wire
protocol server as robustly as possible such that servers can
aggressively cache responses and defer as much server load as
possible.

This commit represents the initial implementation of a general
caching layer in wire protocol version 2.

We define a new interface and behavior for a wire protocol cacher
in repository.py. (This is probably where a reviewer should look
first to understand what is going on.)

The bulk of the added code is in wireprotov2server.py, where we
define how a command can opt in to being cached and integrate
caching into command dispatching.

From a very high-level:

* A command can declare itself as cacheable by providing a callable
  that can be used to derive a cache key.
* At dispatch time, if a command is cacheable, we attempt to
  construct a cacher and use it for serving the request and/or
  caching the request.
* The dispatch layer handles the bulk of the business logic for
  caching, making cachers mostly "dumb content stores."
* The mechanism for invalidating cached entries (one of the harder
  parts about caching in general) is by varying the cache key when
  state changes. As such, cachers don't need to be concerned with
  cache invalidation.

Initially, we've hooked up support for caching "manifestdata" and
"filedata" commands. These are the simplest to cache, as they should
be immutable over time. Caching of commands related to changeset
data is a bit harder (because cache validation is impacted by
changes to bookmarks, phases, etc). This will be implemented later.
(Strictly speaking, censoring a file should invalidate caches. I've
added an inline TODO to track this edge case.)

To prove it works, this commit implements a test-only extension
providing in-memory caching backed by an lrucachedict. A new test
showing this extension behaving properly is added. FWIW, the
cacher is ~50 lines of code, demonstrating the relative ease with
which a cache can be added to a server.

While the test cacher is not suitable for production workloads, just
for kicks I performed a clone of just the changeset and manifest data
for the mozilla-unified repository. With a fully warmed cache (of just
the manifest data since changeset data is not cached), server-side
CPU usage dropped from ~73s to ~28s. That's pretty significant and
demonstrates the potential that response caching has on server
scalability!

Differential Revision: https://phab.mercurial-scm.org/D4773

# All revsets ever used with revsetbenchmarks.py script
#
# The goal of this file is to gather all revsets ever used for benchmarking
# revset's performance. It should be used to gather revsets that test a
# specific usecase or a specific implementation of revset predicates.
# If you are working on the smartset implementation itself, check
# 'base-revsets.txt'.
#
# Please update this file with any revsets you use for benchmarking a change so
# that future contributors can easily find and retest it when doing further
# modification. Feel free to highlight interesting variants if needed.


## Revset from this section are all extracted from changelog when this file was
# created. Feel free to dig and improve documentation.

# Used in revision da05fe01170b
(20000::) - (20000)
# Used in revision 95af98616aa7
parents(20000)
# Used in revision 186fd06283b4
(_intlist('20000\x0020001')) and merge()
# Used in revision 911f5a6579d1
p1(20000)
p2(10000)
# Used in revision b6dc3b79bb25
0::
# Used in revision faf4f63533ff
bookmark()
# Used in revision 22ba2c0825da
tip~25
# Used in revision 0cf46b8298fe
bisect(range)
# Used in revision 5b65429721d5
divergent()
# Used in revision 6261b9c549a2
file(COPYING)
# Used in revision 44f471102f3a
follow(COPYING)
# Used in revision 8040a44aab1c
origin(tip)
# Used in revision bbf4f3dfd700
rev(25)
# Used in revision a428db9ab61d
p1()
# Used in revision c1546d7400ef
min(0::)
# Used in revision 546fa6576815
author(lmoscovicz) or author(mpm)
author(mpm) or author(lmoscovicz)
# Used in revision 9bfe68357c01
public() and id("d82e2223f132")
# Used in revision ba89f7b542c9
rev(25)
# Used in revision eb763217152a
rev(210000)
# Used in revision 69524a05a7fa
10:100
parents(10):parents(100)
# Used in revision 6f1b8b3f12fd
100~5
parents(100)~5
(100~5)~5
# Used in revision 7a42e5d4c418
children(tip~100)
# Used in revision 7e8737e6ab08
100^1
parents(100)^1
(100^1)^1
# Used in revision 30e0dcd7c5ff
matching(100)
matching(parents(100))
# Used in revision aafeaba22826
0|1|2|3|4|5|6|7|8|9
# Used in revision 33c7a94d4dd0
tip:0
# Used in revision 7d369fae098e
(0:100000)
# Used in revision b333ca94403d
0 + 1 + 2 + ... + 200
0 + 1 + 2 + ... + 1000
sort(0 + 1 + 2 + ... + 200)
sort(0 + 1 + 2 + ... + 1000)
# Used in revision 7fbef7932af9
first(0 + 1 + 2 + ... + 1000)
# Used in revision ceaf04bb14ff
0:1000
# Used in revision 262e6ad93885
not public()
(tip~1000::) - public()
not public() and branch("default")
# Used in revision 15412bba5a68
0::tip

## all the revsets from this section have been taken from the former central file
# for revset's benchmarking, they are undocumented for this reason.
all()
draft()
::tip
draft() and ::tip
::tip and draft()
author(lmoscovicz)
author(mpm)
::p1(p1(tip))::
public()
:10000 and public()
:10000 and draft()
(not public() - obsolete())

# The one below is used by rebase
(children(ancestor(tip~5, tip)) and ::(tip~5))::

# those two `roots(...)` inputs are close to what phase movement use.
roots((tip~100::) - (tip~100::tip))
roots((0::) - (0::tip))

# more roots testing
roots(tip~100:)
roots(:42)
roots(not public())
roots((0:tip)::)
roots(0::tip)
42:68 and roots(42:tip)
# Used in revision f140d6207cca
roots(0:tip)
# test disjoint set with multiple roots
roots((:42) + (tip~42:))

# Testing the behavior of "head()" in various situations
head()
head() - public()
draft() and head()
head() and author("mpm")

# testing the mutable phases set
draft()
secret()

# test finding common ancestors
heads(commonancestors(last(head(), 2)))
heads(commonancestors(head()))