annotate hgext/lfs/TODO.rst @ 42278:8dc22a209420

automation: wait for instance profiles and roles

Otherwise there is a race condition between creating the resources
and us attempting to use them / them becoming available.

The role waiter API was recently introduced, so we had to upgrade
the boto3 package to get it. Other packages were also updated
to latest versions just because.

Even with this change, I still run into issues with the IAM instance
profile not being available when we attempt to create an EC2 instance
using a just-created profile. I'm not sure what's going on. Possibly
a bug on Amazon's end. But the new behavior is "more correct."

Differential Revision: https://phab.mercurial-scm.org/D6286

Prior to removing (EXPERIMENTAL)
--------------------------------

These things affect UI and/or behavior, and should probably be implemented (or
ruled out) prior to taking off the experimental shrinkwrap.

#. Finish the `hg convert` story

   * Add an argument to accept a rules file to apply during conversion?
     Currently `lfs.track` is the only way to affect the conversion.
   * drop `lfs.track` config settings
   * splice in `.hglfs` file for normal repo -> lfs conversions?

#. Stop uploading blobs when pushing between local repos

   * Could probably hardlink directly to the other local repo's store
   * Support inferring `lfs.url` for local push/pull (currently only supports
     http)

#. Stop uploading blobs on strip/amend/histedit/etc.

   * This seems to be a side effect of doing it for `hg bundle`, which probably
     makes sense.

#. Handle a server with the extension loaded and a client without the extension
   more gracefully.

   * `changegroup3` is still experimental, and not enabled by default.
   * Figure out how to `introduce LFS to the server repo
     <https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-September/122281.html>`_.
     See the TODO in test-lfs-serve.t.

#. Remove `lfs.retry` hack in client?  This came from FB, but it's not clear why
   it is/was needed.

#. `hg export` currently writes out the LFS blob.  Should it write the pointer
   instead?

   * `hg diff` is similar, and probably shouldn't see the pointer file

#. `Fix https multiplexing, and re-enable workers
   <https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-January/109916.html>`_.

#. Show to-be-applied rules with `hg files -r 'wdir()' 'set:lfs()'`

   * `debugignore` can show file + line number, so a dedicated command could be
     useful too.

#. Filesets, revsets and templates

   * A dedicated revset should be faster than `'file(set:lfs())'`
   * Attach `{lfsoid}` and `{lfspointer}` to `general keywords
     <https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-January/110251.html>`_,
     IFF the file is a blob
   * Drop existing items that would be redundant with general support

#. Can `grep` avoid downloading most things?

   * Add a command option to skip LFS blobs?

#. Add a flag that's visible in `hg files -v` to indicate external storage?

#. Server side issues

   * Check for local disk space before allowing upload.  (I've got a patch for
     this.)
   * Make sure the http codes used are appropriate.
   * `Why is copying the Authorization header into the JSON payload necessary
     <https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-April/116230.html>`_?
   * `LFS-Authenticate` header support in client and server(?)

#. Add locks on cache and blob store

   * This is complicated with a global store, and multiple potentially unrelated
     local repositories that reference the same blob.
   * Alternately, maybe just handle collisions when trying to create the same
     blob in the store somehow.

#. Are proper file sizes reported in `debugupgraderepo`?

#. Finish prefetching files

   * `-T {rawdata}`
   * `verify`
   * `grep`

#. Output cleanup

   * Can we print the url when connecting to the blobstore?  (A sudden
     connection refused after pulling commits looks confusing.)  Problem is,
     'pushing to main url' is printed, and then lfs wants to upload before going
     back to the main repo transfer, so then *that* could be confusing with
     extra output. (This is kinda improved with 380f5131ee7b and 9f78d10742af.)

   * Add more progress indicators?  Uploading a large repo looks idle for a long
     time while it scans for blobs in each outgoing revision.

   * Print filenames instead of hashes in error messages

     * subrepo aware paths, where necessary

   * Is existing output at the right status/note/debug level?

#. Can `verify` be done without downloading everything?

   * If we know that we are talking to an hg server, we can leverage the fact
     that it validates in the Batch API portion, and skip d/l altogether.  OTOH,
     maybe we should download the files unconditionally for forensics.  The
     alternative is to define a custom transfer handler that definitively
     verifies without transferring, and then cache those results.  When verify
     comes looking, look in the cache instead of actually opening the file and
     processing it.

   * Yuya has concerns about when blob fetch takes place vs when revlog is
     verified.  Since the visible hash matches the blob content, I don't think
     there's a way to verify the pointer file that's actually stored in the
     filelog (other than basic JSON checks).  Full verification requires the
     blob.  See
     https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-April/116133.html

   * Opening a corrupt pointer file aborts.  It probably shouldn't for verify.


Future ideas/features/polishing
-------------------------------

These aren't in any particular order, and are things that don't have obvious BC
concerns.

#. Garbage collection `(issue5790) <https://bz.mercurial-scm.org/show_bug.cgi?id=5790>`_

   * This gets complicated because of the global cache, which may or may not
     consist of hardlinks to the repo, and may be in use by other repos.  (So
     the gc may be pointless.)

#. `Compress blobs <https://github.com/git-lfs/git-lfs/issues/260>`_

   * 700MB repo becomes 2.5GB with all lfs blobs
   * What implications are there for filesystem paths that don't indicate
     compression?  (i.e. how to share with global cache and other local repos?)
   * Probably needs to be stored under `.hg/store/lfs/zstd`, with a repo
     requirement.
   * Allow tuneable compression type and settings?
   * Support compression over the wire if both sides understand the compression?
   * `debugupgraderepo` to convert?
   * Probably not worth supporting compressed and uncompressed concurrently

#. Determine things to upload with `readfast()
   <https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-August/121315.html>`_

   * Significantly faster when pushing an entire large repo to http.
   * Causes test changes to fileset and templates; may need both this and
     current methods of lookup.

#. Is a command to download everything needed?  This would allow copying the
   whole to a portable drive.  Currently this can be effected by running
   `hg verify`.

#. Stop reading in entire file into one buffer when passing through filelog
   interface

   * `Requires major replumbing to core
     <https://www.mercurial-scm.org/wiki/HandlingLargeFiles>`_

#. Keep corrupt files around in 'store/lfs/incoming' for forensics?

   * Files should be downloaded to 'incoming', and moved to normal location when
     done.

#. Client side path enhancements

   * Support paths.default:lfs = ... style paths
   * SSH -> https server inference

     * https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-April/115416.html
     * https://github.com/git-lfs/git-lfs/blob/master/docs/api/server-discovery.md#guessing-the-server

#. Server enhancements

   * Add support for transfer quotas?
   * Download should be able to send the file in chunks, without reading the
     whole thing into memory
     (https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-March/114584.html)
   * Support for resuming transfers

#. Handle 3rd party server storage.

   * Teach client to handle lfs `verify` action.  This is needed after the
     server instructs the client to upload the file to another server, in order
     to tell the server that the upload completed.
   * Teach the server to send redirects if configured, and process `verify`
     requests.

#. `Is any hg-git work needed
   <https://groups.google.com/d/msg/hg-git/XYNQuudteeM/ivt8gXoZAAAJ>`_?