view mercurial/help/internals/censor.txt @ 40178:46a40bce3ae0

wireprotov2: define and implement "filesdata" command

Previously, the only way to access file revision data was the
"filedata" command. This command is useful to have. But, it only
allowed resolving revision data for a single file. This meant that
clients needed to send 1 command for each tracked path they were
seeking data on. Furthermore, those commands would need to enumerate
the exact file nodes they wanted data for.

This approach meant that clients were sending a lot of data to
remotes in order to request file data. e.g. if there were 1M
file revisions, we'd need at least 20,000,000 bytes just to encode
file nodes! Many clients on the internet don't have that kind of
upload capacity.

In order to limit the amount of data that clients must send, we'll
need more efficient ways to request repository data.

This commit defines and implements a new "filesdata" command. This
command allows the retrieval of data for multiple files by specifying
changeset revisions and optional file patterns. The command figures
out what file revisions are "relevant" and sends them in bulk.

The logic around choosing which file revisions to send in the case of
haveparents not being set is overly simple and will over-send files. We
will need more smarts here eventually. (Specifically, the client will
need to tell the server which revisions it knows about.) This work
is deferred until a later time.

Differential Revision: https://phab.mercurial-scm.org/D4981

The censor system allows retroactively removing content from
files. Actually censoring a node requires using the censor extension,
but the functionality for handling censored nodes is partially in core.

Censored nodes in a filelog have the flag ``REVIDX_ISCENSORED`` set,
and the contents of the censored node are replaced with a censor
tombstone. For historical reasons, the tombstone is packed in the
filelog metadata field ``censored``. This allows censored nodes to be
(mostly) safely transmitted through old formats like changegroup
versions 1 and 2. When using changegroup formats older than 3, the
receiver is required to re-add the ``REVIDX_ISCENSORED`` flag when
storing the revision. This depends on the ``censored`` metadata key
never being used for anything other than censoring revisions, which is
true as of January 2017. Note that the revlog flag is the
authoritative marker of a censored node: the tombstone should only be
consulted when looking for a reason a node was censored or when revlog
flags are unavailable as mentioned above.

The tombstone data is a free-form string. It's expected that users of
censor will want to record the reason for censoring a node in the
tombstone. Censored nodes must be able to fit in the size of the
content being censored.