comparison mercurial/streamclone.py @ 50666:60f9602b413e

clonebundles: add support for inline (streaming) clonebundles

The idea behind inline clonebundles is to send them through
the ssh or https connection to the Mercurial server.
We've been using this specifically for streaming clonebundles,
although it works for 'regular' clonebundles as well
(but is less relevant, since pullbundles exist).

We've had this enabled for around 9 months for a part
of our users.

A few benefits are:
- no need to secure an external system,
  since everything goes through the same Mercurial server
- easier scaling (in our case: no risk of inconsistencies
  between multiple mercurial-server mirrors and nginx clonebundles hosts)

Remaining topics/questions right now:
- The inline clonebundles don't work for https yet.
  This is because httppeer doesn't seem to support sending client
  capabilities.
  I didn't focus on that as my main goal was to get this working
  for ssh.

50646:c814101560d9

        with repo.transaction(b'clone'):
            with repo.svfs.backgroundclosing(repo.ui, expectedcount=filecount):
                for i in range(filecount):
                    # XXX doesn't support '\n' or '\r' in filenames
                    l = fp.readline()
                    try:
                        name, size = l.split(b'\0', 1)
                        size = int(size)
                    except (ValueError, TypeError):
                        raise error.ResponseError(

50666:60f9602b413e

        with repo.transaction(b'clone'):
            with repo.svfs.backgroundclosing(repo.ui, expectedcount=filecount):
                for i in range(filecount):
                    # XXX doesn't support '\n' or '\r' in filenames
                    if util.safehasattr(fp, 'readline'):
                        l = fp.readline()
                    else:
                        # inline clonebundles use a chunkbuffer, so no readline
                        # --> this should be small anyway, the first line
                        # only contains the size of the bundle
                        l_buf = []
                        while not (l_buf and l_buf[-1] == b'\n'):
                            l_buf.append(fp.read(1))
                        l = b''.join(l_buf)
                    try:
                        name, size = l.split(b'\0', 1)
                        size = int(size)
                    except (ValueError, TypeError):
                        raise error.ResponseError(