mercurial/help/subrepos.txt

protocol: send application/mercurial-0.2 responses to capable clients

With this commit, the HTTP transport now parses the X-HgProto-<N>
header to determine what media type and compression engine to use for
responses. So far, we only compress responses that are already being
compressed with zlib today (stream response types to specific
commands). We can expand things to cover additional response types
later.

The practical side-effect of this commit is that non-zlib compression
engines will be used if both ends support them. This means if both
ends have zstd support, zstd - not zlib - will be used to compress
data!

When cloning the mozilla-unified repository between a local HTTP
server and client, the benefits of non-zlib compression are quite
noticeable:

  engine     server CPU (s)   client CPU (s)    bundle size
zlib (l=6)      174.1            283.2         1,148,547,026
zstd (l=1)       99.2            267.3         1,127,513,841
zstd (l=3)      103.1            266.9         1,018,861,363
zstd (l=7)      128.3            269.7           919,190,278
zstd (l=10)     162.0               -            894,547,179
none             95.3            277.2         4,097,566,064

The default zstd compression level is 3. So if you deploy zstd
capable Mercurial to your clients and servers and CPU time on
your server is dominated by "getbundle" requests (clients cloning
and pulling) - and my experience at Mozilla tells me this is often
the case - this commit could drastically reduce your server-side
CPU usage *and* save on bandwidth costs!

Another benefit of this change is that server operators can install
*any* compression engine. While it isn't enabled by default, the
"none" compression engine can now be used to disable wire protocol
compression completely. Previously, commands like "getbundle" always
zlib compressed output, adding considerable overhead to generating
responses. If you are on a high speed network and your server is under
high load, it might be advantageous to trade bandwidth for CPU.
Although, zstd at level 1 doesn't use that much CPU, so I'm not
convinced that disabling compression wholesale is worthwhile. And, my
data seems to indicate a slow down on the client without compression.
I suspect this is due to a lack of buffering resulting in an increase
in socket read() calls and/or the fact we're transferring an extra 3 GB
of data (parsing HTTP chunked transfer and processing extra TCP packets
can add up). This is definitely worth investigating and optimizing. But
since the "none" compressor isn't enabled by default, I'm inclined to
punt on this issue.

This commit introduces tons of tests. Some of these should arguably
have been implemented on previous commits. But it was difficult to
test without the server functionality in place.

Subrepositories let you nest external repositories or projects into a
parent Mercurial repository, and make commands operate on them as a
group.

Mercurial currently supports Mercurial, Git, and Subversion
subrepositories.

Subrepositories are made of three components:

1. Nested repository checkouts. They can appear anywhere in the
   parent working directory.

2. Nested repository references. They are defined in ``.hgsub``, which
   should be placed in the root of working directory, and
   tell where the subrepository checkouts come from. Mercurial
   subrepositories are referenced like::

     path/to/nested = https://example.com/nested/repo/path

   Git and Subversion subrepos are also supported::

     path/to/nested = [git]git://example.com/nested/repo/path
     path/to/nested = [svn]https://example.com/nested/trunk/path

   where ``path/to/nested`` is the checkout location relatively to the
   parent Mercurial root, and ``https://example.com/nested/repo/path``
   is the source repository path. The source can also reference a
   filesystem path.

   Note that ``.hgsub`` does not exist by default in Mercurial
   repositories, you have to create and add it to the parent
   repository before using subrepositories.

3. Nested repository states. They are defined in ``.hgsubstate``, which
   is placed in the root of working directory, and
   capture whatever information is required to restore the
   subrepositories to the state they were committed in a parent
   repository changeset. Mercurial automatically record the nested
   repositories states when committing in the parent repository.

   .. note::

      The ``.hgsubstate`` file should not be edited manually.


Adding a Subrepository
======================

If ``.hgsub`` does not exist, create it and add it to the parent
repository. Clone or checkout the external projects where you want it
to live in the parent repository. Edit ``.hgsub`` and add the
subrepository entry as described above. At this point, the
subrepository is tracked and the next commit will record its state in
``.hgsubstate`` and bind it to the committed changeset.

Synchronizing a Subrepository
=============================

Subrepos do not automatically track the latest changeset of their
sources. Instead, they are updated to the changeset that corresponds
with the changeset checked out in the top-level changeset. This is so
developers always get a consistent set of compatible code and
libraries when they update.

Thus, updating subrepos is a manual process. Simply check out target
subrepo at the desired revision, test in the top-level repo, then
commit in the parent repository to record the new combination.

Deleting a Subrepository
========================

To remove a subrepository from the parent repository, delete its
reference from ``.hgsub``, then remove its files.

Interaction with Mercurial Commands
===================================

:add: add does not recurse in subrepos unless -S/--subrepos is
    specified.  However, if you specify the full path of a file in a
    subrepo, it will be added even without -S/--subrepos specified.
    Subversion subrepositories are currently silently
    ignored.

:addremove: addremove does not recurse into subrepos unless
    -S/--subrepos is specified.  However, if you specify the full
    path of a directory in a subrepo, addremove will be performed on
    it even without -S/--subrepos being specified.  Git and
    Subversion subrepositories will print a warning and continue.

:archive: archive does not recurse in subrepositories unless
    -S/--subrepos is specified.

:cat: cat currently only handles exact file matches in subrepos.
    Subversion subrepositories are currently ignored.

:commit: commit creates a consistent snapshot of the state of the
    entire project and its subrepositories. If any subrepositories
    have been modified, Mercurial will abort.  Mercurial can be made
    to instead commit all modified subrepositories by specifying
    -S/--subrepos, or setting "ui.commitsubrepos=True" in a
    configuration file (see :hg:`help config`).  After there are no
    longer any modified subrepositories, it records their state and
    finally commits it in the parent repository.  The --addremove
    option also honors the -S/--subrepos option.  However, Git and
    Subversion subrepositories will print a warning and abort.

:diff: diff does not recurse in subrepos unless -S/--subrepos is
    specified. Changes are displayed as usual, on the subrepositories
    elements. Subversion subrepositories are currently silently ignored.

:files: files does not recurse into subrepos unless -S/--subrepos is
    specified.  However, if you specify the full path of a file or
    directory in a subrepo, it will be displayed even without
    -S/--subrepos being specified.  Git and Subversion subrepositories
    are currently silently ignored.

:forget: forget currently only handles exact file matches in subrepos.
    Git and Subversion subrepositories are currently silently ignored.

:incoming: incoming does not recurse in subrepos unless -S/--subrepos
    is specified. Git and Subversion subrepositories are currently
    silently ignored.

:outgoing: outgoing does not recurse in subrepos unless -S/--subrepos
    is specified. Git and Subversion subrepositories are currently
    silently ignored.

:pull: pull is not recursive since it is not clear what to pull prior
    to running :hg:`update`. Listing and retrieving all
    subrepositories changes referenced by the parent repository pulled
    changesets is expensive at best, impossible in the Subversion
    case.

:push: Mercurial will automatically push all subrepositories first
    when the parent repository is being pushed. This ensures new
    subrepository changes are available when referenced by top-level
    repositories.  Push is a no-op for Subversion subrepositories.

:status: status does not recurse into subrepositories unless
    -S/--subrepos is specified. Subrepository changes are displayed as
    regular Mercurial changes on the subrepository
    elements. Subversion subrepositories are currently silently
    ignored.

:remove: remove does not recurse into subrepositories unless
    -S/--subrepos is specified.  However, if you specify a file or
    directory path in a subrepo, it will be removed even without
    -S/--subrepos.  Git and Subversion subrepositories are currently
    silently ignored.

:update: update restores the subrepos in the state they were
    originally committed in target changeset. If the recorded
    changeset is not available in the current subrepository, Mercurial
    will pull it in first before updating.  This means that updating
    can require network access when using subrepositories.

Remapping Subrepositories Sources
=================================

A subrepository source location may change during a project life,
invalidating references stored in the parent repository history. To
fix this, rewriting rules can be defined in parent repository ``hgrc``
file or in Mercurial configuration. See the ``[subpaths]`` section in
hgrc(5) for more details.