clonebundles: optional memory-requirement attribution

clonebundles: optional memory-requirement attribution

The new REQUIREDRAM option allows a client to skip bundles it isn't
expected to handle well, e.g. without swapping. This allows a fallback
path to be provided e.g. using zstd level 10 instead of 22.

Differential Revision: https://phab.mercurial-scm.org/D8645

--- a/mercurial/exchange.py	Tue Jun 09 11:22:31 2020 +0200
+++ b/mercurial/exchange.py	Mon Jun 22 00:00:07 2020 +0200
@@ -3025,6 +3025,23 @@
             )
             continue
 
+        if b'REQUIREDRAM' in entry:
+            try:
+                requiredram = util.sizetoint(entry[b'REQUIREDRAM'])
+            except error.ParseError:
+                repo.ui.debug(
+                    b'filtering %s due to a bad REQUIREDRAM attribute\n'
+                    % entry[b'URL']
+                )
+                continue
+            actualram = repo.ui.estimatememory()
+            if actualram is not None and actualram * 0.66 < requiredram:
+                repo.ui.debug(
+                    b'filtering %s as it needs more than 2/3 of system memory\n'
+                    % entry[b'URL']
+                )
+                continue
+
         newentries.append(entry)
 
     return newentries

--- a/relnotes/next	Tue Jun 09 11:22:31 2020 +0200
+++ b/relnotes/next	Mon Jun 22 00:00:07 2020 +0200
@@ -1,5 +1,9 @@
 == New Features ==
 
+ * clonebundles can be annotated with the expected memory requirements
+   using the `REQUIREDRAM` option. This allows clients to skip
+   bundles created with large zstd windows and fallback to larger, but
+   less demanding bundles.
 
 == New Experimental Features ==
 

--- a/tests/test-clonebundles.t	Tue Jun 09 11:22:31 2020 +0200
+++ b/tests/test-clonebundles.t	Mon Jun 22 00:00:07 2020 +0200
@@ -551,3 +551,88 @@
   searching for changes
   no changes found
   2 local changesets published
+  $ killdaemons.py
+
+A manifest with a gzip bundle requiring too much memory for a 16MB system and working
+on a 32MB system.
+
+  $ "$PYTHON" $TESTDIR/dumbhttp.py -p $HGPORT1 --pid http.pid
+  $ cat http.pid >> $DAEMON_PIDS
+  $ hg -R server serve -d -p $HGPORT --pid-file hg.pid --accesslog access.log
+  $ cat hg.pid >> $DAEMON_PIDS
+
+  $ cat > server/.hg/clonebundles.manifest << EOF
+  > http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2 REQUIREDRAM=12MB
+  > EOF
+
+  $ hg clone -U --debug --config ui.available-memory=16MB http://localhost:$HGPORT gzip-too-large
+  using http://localhost:$HGPORT/
+  sending capabilities command
+  sending clonebundles command
+  filtering http://localhost:$HGPORT1/gz-a.hg as it needs more than 2/3 of system memory
+  no compatible clone bundles available on server; falling back to regular clone
+  (you may want to report this to the server operator)
+  query 1; heads
+  sending batch command
+  requesting all changes
+  sending getbundle command
+  bundle2-input-bundle: with-transaction
+  bundle2-input-part: "changegroup" (params: 1 mandatory 1 advisory) supported
+  adding changesets
+  add changeset 53245c60e682
+  add changeset aaff8d2ffbbf
+  adding manifests
+  adding file changes
+  adding bar revisions
+  adding foo revisions
+  bundle2-input-part: total payload size 920
+  bundle2-input-part: "listkeys" (params: 1 mandatory) supported
+  bundle2-input-part: "phase-heads" supported
+  bundle2-input-part: total payload size 24
+  bundle2-input-part: "cache:rev-branch-cache" (advisory) supported
+  bundle2-input-part: total payload size 59
+  bundle2-input-bundle: 4 parts total
+  checking for updated bookmarks
+  updating the branch cache
+  added 2 changesets with 2 changes to 2 files
+  new changesets 53245c60e682:aaff8d2ffbbf
+  calling hook changegroup.lfiles: hgext.largefiles.reposetup.checkrequireslfiles
+  (sent 4 HTTP requests and * bytes; received * bytes in responses) (glob)
+
+  $ hg clone -U --debug --config ui.available-memory=32MB http://localhost:$HGPORT gzip-too-large2
+  using http://localhost:$HGPORT/
+  sending capabilities command
+  sending clonebundles command
+  applying clone bundle from http://localhost:$HGPORT1/gz-a.hg
+  bundle2-input-bundle: 1 params with-transaction
+  bundle2-input-part: "changegroup" (params: 1 mandatory 1 advisory) supported
+  adding changesets
+  add changeset 53245c60e682
+  add changeset aaff8d2ffbbf
+  adding manifests
+  adding file changes
+  adding bar revisions
+  adding foo revisions
+  bundle2-input-part: total payload size 920
+  bundle2-input-part: "cache:rev-branch-cache" (advisory) supported
+  bundle2-input-part: total payload size 59
+  bundle2-input-bundle: 2 parts total
+  updating the branch cache
+  added 2 changesets with 2 changes to 2 files
+  finished applying clone bundle
+  query 1; heads
+  sending batch command
+  searching for changes
+  all remote heads known locally
+  no changes found
+  sending getbundle command
+  bundle2-input-bundle: with-transaction
+  bundle2-input-part: "listkeys" (params: 1 mandatory) supported
+  bundle2-input-part: "phase-heads" supported
+  bundle2-input-part: total payload size 24
+  bundle2-input-bundle: 2 parts total
+  checking for updated bookmarks
+  2 local changesets published
+  calling hook changegroup.lfiles: hgext.largefiles.reposetup.checkrequireslfiles
+  (sent 4 HTTP requests and * bytes; received * bytes in responses) (glob)
+  $ killdaemons.py