annotate mercurial/help/internals/wireprotocolv2.txt @ 40172:30f70d11c224

wireprotov2: advertise recommended batch size for requests

Currently, exchangev2 hardcodes the batch size for how many revisions
to fetch per command request. A single value is not appropriate
for every repository because some repositories may have a drastically
different "shape" from other repositories. e.g. a repo with lots of
small files may benefit from larger batch sizes than a repo with lots
of large files. And depending on caching used by the server, the server
may wish to control the number of commands (to e.g. mitigate overhead
of following content redirects).

This commit teaches wireprotov2 commands to declare extra metadata
which is advertised as part of the command descriptor. The manifestdata
command has been taught to advertise a recommended batch size for
requests.

Differential Revision: https://phab.mercurial-scm.org/D4975

**Experimental and under active development**

This section documents the wire protocol commands exposed to transports
using the frame-based protocol. The set of commands exposed through
these transports is distinct from the set of commands exposed to legacy
transports.

The frame-based protocol uses CBOR to encode command execution requests.
All command arguments must be mapped to a specific or set of CBOR data
types.

The response to many commands is also CBOR. There is no common response
format: each command defines its own response format.

TODOs
=====

* Add "node namespace" support to each command. In order to support
  SHA-1 hash transition, we want servers to be able to expose different
  "node namespaces" for the same data. Every command operating on nodes
  should specify which "node namespace" it is operating on and responses
  should encode the "node namespace" accordingly.

Commands
========

The sections below detail all commands available to wire protocol version
2.

branchmap
---------

Obtain heads in named branches.

Receives no arguments.

The response is a map with bytestring keys defining the branch name.
Values are arrays of bytestring defining raw changeset nodes.

capabilities
------------

Obtain the server's capabilities.

Receives no arguments.

This command is typically called only as part of the handshake during
initial connection establishment.

The response is a map with bytestring keys defining server information.

The defined keys are:

commands
   A map defining available wire protocol commands on this server.

   Keys in the map are the names of commands that can be invoked. Values
   are maps defining information about that command. The bytestring keys
   are:

      args
         (map) Describes arguments accepted by the command.

         Keys are bytestrings denoting the argument name.

         Values are maps describing the argument. The map has the following
         bytestring keys:

         default
            (varied) The default value for this argument if not specified. Only
            present if ``required`` is not true.

         required
            (boolean) Whether the argument must be specified. Failure to send
            required arguments will result in an error executing the command.

         type
            (bytestring) The type of the argument. e.g. ``bytes`` or ``bool``.

         validvalues
            (set) Values that are recognized for this argument. Some arguments
            only allow a fixed set of values to be specified. These arguments
            may advertise that set in this key. If this set is advertised and
            a value not in this set is specified, the command should result
            in error.

      permissions
         An array of permissions required to execute this command.

      *
         (various) Individual commands may define extra keys that supplement
         generic command metadata. See the command definition for more.

framingmediatypes
   An array of bytestrings defining the supported framing protocol
   media types. Servers will not accept media types not in this list.

pathfilterprefixes
   (set of bytestring) Matcher prefixes that are recognized when performing
   path filtering. Specifying a path filter whose type/prefix does not
   match one in this set will likely be rejected by the server.

rawrepoformats
   An array of storage formats the repository is using. This set of
   requirements can be used to determine whether a client can read a
   *raw* copy of file data available.

redirect
   A map declaring potential *content redirects* that may be used by this
   server. Contains the following bytestring keys:

   targets
      (array of maps) Potential redirect targets. Values are maps describing
      this target in more detail. Each map has the following bytestring keys:

      name
         (bytestring) Identifier for this target. The identifier will be used
         by clients to uniquely identify this target.

      protocol
         (bytestring) High-level network protocol. Values can be
         ``http``, ```https``, ``ssh``, etc.

      uris
          (array of bytestrings) Representative URIs for this target.

      snirequired (optional)
          (boolean) Indicates whether Server Name Indication is required
          to use this target. Defaults to False.

      tlsversions (optional)
          (array of bytestring) Indicates which TLS versions are supported by
          this target. Values are ``1.1``, ``1.2``, ``1.3``, etc.

   hashes
      (array of bytestring) Indicates support for hashing algorithms that are
      used to ensure content integrity. Values include ``sha1``, ``sha256``,
      etc.

changesetdata
-------------

Obtain various data related to changesets.

The command accepts the following arguments:

noderange
   (array of arrays of bytestrings) An array of 2 elements, each being an
   array of node bytestrings. The first array denotes the changelog revisions
   that are already known to the client. The second array denotes the changelog
   revision DAG heads to fetch. The argument essentially defines a DAG range
   bounded by root and head nodes to fetch.

   The roots array may be empty. The heads array must be defined.

nodes
   (array of bytestrings) Changelog revisions to request explicitly.

nodesdepth
   (unsigned integer) Number of ancestor revisions of elements in ``nodes``
   to also fetch. When defined, for each element in ``nodes``, DAG ancestors
   will be walked until at most N total revisions are emitted.

fields
   (set of bytestring) Which data associated with changelog revisions to
   fetch. The following values are recognized:

   bookmarks
      Bookmarks associated with a revision.

   parents
      Parent revisions.

   phase
      The phase state of a revision.

   revision
      The raw, revision data for the changelog entry. The hash of this data
      will match the revision's node value.

The server resolves the set of revisions relevant to the request by taking
the union of the ``noderange`` and ``nodes`` arguments. At least one of these
arguments must be defined.

The response bytestream starts with a CBOR map describing the data that follows.
This map has the following bytestring keys:

totalitems
   (unsigned integer) Total number of changelog revisions whose data is being
   transferred. This maps to the set of revisions in the requested node
   range, not the total number of records that follow (see below for why).

Following the map header is a series of 0 or more CBOR values. If values
are present, the first value will always be a map describing a single changeset
revision.

If the ``fieldsfollowing`` key is present, the map will immediately be followed
by N CBOR bytestring values, where N is the number of elements in
``fieldsfollowing``. Each bytestring value corresponds to a field denoted
by ``fieldsfollowing``.

Following the optional bytestring field values is the next revision descriptor
map, or end of stream.

Each revision descriptor map has the following bytestring keys:

node
   (bytestring) The node value for this revision. This is the SHA-1 hash of
   the raw revision data.

bookmarks (optional)
   (array of bytestrings) Bookmarks attached to this revision. Only present
   if ``bookmarks`` data is being requested and the revision has bookmarks
   attached.

fieldsfollowing (optional)
   (array of 2-array) Denotes what fields immediately follow this map. Each
   value is an array with 2 elements: the bytestring field name and an unsigned
   integer describing the length of the data, in bytes.

   If this key isn't present, no special fields will follow this map.

   The following fields may be present:

   revision
      Raw, revision data for the changelog entry. Contains a serialized form
      of the changeset data, including the author, date, commit message, set
      of changed files, manifest node, and other metadata.

      Only present if the ``revision`` field was requested.

parents (optional)
   (array of bytestrings) The nodes representing the parent revisions of this
   revision. Only present if ``parents`` data is being requested.

phase (optional)
   (bytestring) The phase that a revision is in. Recognized values are
   ``secret``, ``draft``, and ``public``. Only present if ``phase`` data
   is being requested.

If nodes are requested via ``noderange``, they will be emitted in DAG order,
parents always before children.

If nodes are requested via ``nodes``, they will be emitted in requested order.

Nodes from ``nodes`` are emitted before nodes from ``noderange``.

The set of changeset revisions emitted may not match the exact set of
changesets requested. Furthermore, the set of keys present on each
map may vary. This is to facilitate emitting changeset updates as well
as new revisions.

For example, if the request wants ``phase`` and ``revision`` data,
the response may contain entries for each changeset in the common nodes
set with the ``phase`` key and without the ``revision`` key in order
to reflect a phase-only update.

TODO support different revision selection mechanisms (e.g. non-public, specific
revisions)
TODO support different hash "namespaces" for revisions (e.g. sha-1 versus other)
TODO support emitting obsolescence data
TODO support filtering based on relevant paths (narrow clone)
TODO support hgtagsfnodes cache / tags data
TODO support branch heads cache
TODO consider unify query mechanism. e.g. as an array of "query descriptors"
rather than a set of top-level arguments that have semantics when combined.

filedata
--------

Obtain various data related to an individual tracked file.

The command accepts the following arguments:

fields
   (set of bytestring) Which data associated with a file to fetch.
   The following values are recognized:

   parents
      Parent nodes for the revision.

   revision
      The raw revision data for a file.

haveparents
   (bool) Whether the client has the parent revisions of all requested
   nodes. If set, the server may emit revision data as deltas against
   any parent revision. If not set, the server MUST only emit deltas for
   revisions previously emitted by this command.

   False is assumed in the absence of any value.

nodes
   (array of bytestrings) File nodes whose data to retrieve.

path
   (bytestring) Path of the tracked file whose data to retrieve.

TODO allow specifying revisions via alternate means (such as from
changeset revisions or ranges)

The response bytestream starts with a CBOR map describing the data that
follows. It has the following bytestream keys:

totalitems
   (unsigned integer) Total number of file revisions whose data is
   being returned.

Following the map header is a series of 0 or more CBOR values. If values
are present, the first value will always be a map describing a single changeset
revision.

If the ``fieldsfollowing`` key is present, the map will immediately be followed
by N CBOR bytestring values, where N is the number of elements in
``fieldsfollowing``. Each bytestring value corresponds to a field denoted
by ``fieldsfollowing``.

Following the optional bytestring field values is the next revision descriptor
map, or end of stream.

Each revision descriptor map has the following bytestring keys:

Each map has the following bytestring keys:

node
   (bytestring) The node of the file revision whose data is represented.

deltabasenode
   (bytestring) Node of the file revision the following delta is against.

   Only present if the ``revision`` field is requested and delta data
   follows this map.

fieldsfollowing
   (array of 2-array) Denotes extra bytestring fields that following this map.
   See the documentation for ``changesetdata`` for semantics.

   The following named fields may be present:

   ``delta``
      The delta data to use to construct the fulltext revision.

      Only present if the ``revision`` field is requested and a delta is
      being emitted. The ``deltabasenode`` top-level key will also be
      present if this field is being emitted.

   ``revision``
      The fulltext revision data for this manifest. Only present if the
      ``revision`` field is requested and a fulltext revision is being emitted.

parents
   (array of bytestring) The nodes of the parents of this file revision.

   Only present if the ``parents`` field is requested.

When ``revision`` data is requested, the server chooses to emit either fulltext
revision data or a delta. What the server decides can be inferred by looking
for the presence of the ``delta`` or ``revision`` keys in the
``fieldsfollowing`` array.

heads
-----

Obtain DAG heads in the repository.

The command accepts the following arguments:

publiconly (optional)
   (boolean) If set, operate on the DAG for public phase changesets only.
   Non-public (i.e. draft) phase DAG heads will not be returned.

The response is a CBOR array of bytestrings defining changeset nodes
of DAG heads. The array can be empty if the repository is empty or no
changesets satisfied the request.

TODO consider exposing phase of heads in response

known
-----

Determine whether a series of changeset nodes is known to the server.

The command accepts the following arguments:

nodes
   (array of bytestrings) List of changeset nodes whose presence to
   query.

The response is a bytestring where each byte contains a 0 or 1 for the
corresponding requested node at the same index.

TODO use a bit array for even more compact response

listkeys
--------

List values in a specified ``pushkey`` namespace.

The command receives the following arguments:

namespace
   (bytestring) Pushkey namespace to query.

The response is a map with bytestring keys and values.

TODO consider using binary to represent nodes in certain pushkey namespaces.

lookup
------

Try to resolve a value to a changeset revision.

Unlike ``known`` which operates on changeset nodes, lookup operates on
node fragments and other names that a user may use.

The command receives the following arguments:

key
   (bytestring) Value to try to resolve.

On success, returns a bytestring containing the resolved node.

manifestdata
------------

Obtain various data related to manifests (which are lists of files in
a revision).

The command accepts the following arguments:

fields
   (set of bytestring) Which data associated with manifests to fetch.
   The following values are recognized:

   parents
      Parent nodes for the manifest.

   revision
      The raw revision data for the manifest.

haveparents
   (bool) Whether the client has the parent revisions of all requested
   nodes. If set, the server may emit revision data as deltas against
   any parent revision. If not set, the server MUST only emit deltas for
   revisions previously emitted by this command.

   False is assumed in the absence of any value.

nodes
   (array of bytestring) Manifest nodes whose data to retrieve.

tree
   (bytestring) Path to manifest to retrieve. The empty bytestring represents
   the root manifest. All other values represent directories/trees within
   the repository.

TODO allow specifying revisions via alternate means (such as from changeset
revisions or ranges)
TODO consider recursive expansion of manifests (with path filtering for
narrow use cases)

The response bytestream starts with a CBOR map describing the data that
follows. It has the following bytestring keys:

totalitems
   (unsigned integer) Total number of manifest revisions whose data is
   being returned.

Following the map header is a series of 0 or more CBOR values. If values
are present, the first value will always be a map describing a single manifest
revision.

If the ``fieldsfollowing`` key is present, the map will immediately be followed
by N CBOR bytestring values, where N is the number of elements in
``fieldsfollowing``. Each bytestring value corresponds to a field denoted
by ``fieldsfollowing``.

Following the optional bytestring field values is the next revision descriptor
map, or end of stream.

Each revision descriptor map has the following bytestring keys:

node
   (bytestring) The node of the manifest revision whose data is represented.

deltabasenode
   (bytestring) The node that the delta representation of this revision is
   computed against. Only present if the ``revision`` field is requested and
   a delta is being emitted.

fieldsfollowing
   (array of 2-array) Denotes extra bytestring fields that following this map.
   See the documentation for ``changesetdata`` for semantics.

   The following named fields may be present:

   ``delta``
      The delta data to use to construct the fulltext revision.

      Only present if the ``revision`` field is requested and a delta is
      being emitted. The ``deltabasenode`` top-level key will also be
      present if this field is being emitted.

   ``revision``
      The fulltext revision data for this manifest. Only present if the
      ``revision`` field is requested and a fulltext revision is being emitted.

parents
   (array of bytestring) The nodes of the parents of this manifest revision.
   Only present if the ``parents`` field is requested.

When ``revision`` data is requested, the server chooses to emit either fulltext
revision data or a delta. What the server decides can be inferred by looking
for the presence of ``delta`` or ``revision`` in the ``fieldsfollowing`` array.

Servers MAY advertise the following extra fields in the capabilities
descriptor for this command:

recommendedbatchsize
   (unsigned integer) Number of revisions the server recommends as a batch
   query size. If defined, clients needing to issue multiple ``manifestdata``
   commands to obtain needed data SHOULD construct their commands to have
   this many revisions per request.

pushkey
-------

Set a value using the ``pushkey`` protocol.

The command receives the following arguments:

namespace
   (bytestring) Pushkey namespace to operate on.
key
   (bytestring) The pushkey key to set.
old
   (bytestring) Old value for this key.
new
   (bytestring) New value for this key.

TODO consider using binary to represent nodes is certain pushkey namespaces.
TODO better define response type and meaning.