exchange: support sorting URLs by client-side preferences
Not all bundles are appropriate for all clients. For example, someone
with a slow Internet connection may want to prefer bz2 bundles over gzip
bundles because they are smaller and don't take as long to transfer.
This is information that a server cannot know on its own. So, we invent
a mechanism for "preferring" server-advertised URLs based on their
attributes.
We could invent a negotiation between client and server where the client
sends its preferences and the sorting/filtering is done server-side.
However, this feels complex. We can avoid complicating the wire protocol
and exposing ourselves to backwards compatible concerns by performing
the sorting locally.
This patch defines a new config option for expressing preferred
attributes in server-advertised bundles.
At Mozilla, we leverage this feature so clients in fast data centers
prefer uncompressed bundles. (We advertise gzip bundles first because
that is a reasonable default.)
I consider this an advanced feature. I'm on the fence as to whether it
should be documented in `hg help config`.
--- a/mercurial/exchange.py Tue Oct 13 12:31:19 2015 -0700
+++ b/mercurial/exchange.py Tue Oct 13 12:30:39 2015 -0700
@@ -1622,7 +1622,7 @@
'operator)\n'))
return
- # TODO sort entries by user preferences.
+ entries = sortclonebundleentries(repo.ui, entries)
url = entries[0]['URL']
repo.ui.status(_('applying clone bundle from %s\n') % url)
@@ -1700,6 +1700,51 @@
return newentries
+def sortclonebundleentries(ui, entries):
+ # experimental config: experimental.clonebundleprefers
+ prefers = ui.configlist('experimental', 'clonebundleprefers', default=[])
+ if not prefers:
+ return list(entries)
+
+ prefers = [p.split('=', 1) for p in prefers]
+
+ # Our sort function.
+ def compareentry(a, b):
+ for prefkey, prefvalue in prefers:
+ avalue = a.get(prefkey)
+ bvalue = b.get(prefkey)
+
+ # Special case for b missing attribute and a matches exactly.
+ if avalue is not None and bvalue is None and avalue == prefvalue:
+ return -1
+
+ # Special case for a missing attribute and b matches exactly.
+ if bvalue is not None and avalue is None and bvalue == prefvalue:
+ return 1
+
+ # We can't compare unless attribute present on both.
+ if avalue is None or bvalue is None:
+ continue
+
+ # Same values should fall back to next attribute.
+ if avalue == bvalue:
+ continue
+
+ # Exact matches come first.
+ if avalue == prefvalue:
+ return -1
+ if bvalue == prefvalue:
+ return 1
+
+ # Fall back to next attribute.
+ continue
+
+ # If we got here we couldn't sort by attributes and prefers. Fall
+ # back to index order.
+ return 0
+
+ return sorted(entries, cmp=compareentry)
+
def trypullbundlefromurl(ui, repo, url):
"""Attempt to apply a bundle from a URL."""
lock = repo.lock()
--- a/tests/test-clonebundles.t Tue Oct 13 12:31:19 2015 -0700
+++ b/tests/test-clonebundles.t Tue Oct 13 12:30:39 2015 -0700
@@ -261,3 +261,97 @@
searching for changes
no changes found
#endif
+
+Set up manifest for testing preferences
+(Remember, the TYPE does not have to match reality - the URL is
+important)
+
+ $ cp full.hg gz-a.hg
+ $ cp full.hg gz-b.hg
+ $ cp full.hg bz2-a.hg
+ $ cp full.hg bz2-b.hg
+ $ cat > server/.hg/clonebundles.manifest << EOF
+ > http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2 extra=a
+ > http://localhost:$HGPORT1/bz2-a.hg BUNDLESPEC=bzip2-v2 extra=a
+ > http://localhost:$HGPORT1/gz-b.hg BUNDLESPEC=gzip-v2 extra=b
+ > http://localhost:$HGPORT1/bz2-b.hg BUNDLESPEC=bzip2-v2 extra=b
+ > EOF
+
+Preferring an undefined attribute will take first entry
+
+ $ hg --config experimental.clonebundleprefers=foo=bar clone -U http://localhost:$HGPORT prefer-foo
+ applying clone bundle from http://localhost:$HGPORT1/gz-a.hg
+ adding changesets
+ adding manifests
+ adding file changes
+ added 2 changesets with 2 changes to 2 files
+ finished applying clone bundle
+ searching for changes
+ no changes found
+
+Preferring bz2 type will download first entry of that type
+
+ $ hg --config experimental.clonebundleprefers=COMPRESSION=bzip2 clone -U http://localhost:$HGPORT prefer-bz
+ applying clone bundle from http://localhost:$HGPORT1/bz2-a.hg
+ adding changesets
+ adding manifests
+ adding file changes
+ added 2 changesets with 2 changes to 2 files
+ finished applying clone bundle
+ searching for changes
+ no changes found
+
+Preferring multiple values of an option works
+
+ $ hg --config experimental.clonebundleprefers=COMPRESSION=unknown,COMPRESSION=bzip2 clone -U http://localhost:$HGPORT prefer-multiple-bz
+ applying clone bundle from http://localhost:$HGPORT1/bz2-a.hg
+ adding changesets
+ adding manifests
+ adding file changes
+ added 2 changesets with 2 changes to 2 files
+ finished applying clone bundle
+ searching for changes
+ no changes found
+
+Sorting multiple values should get us back to original first entry
+
+ $ hg --config experimental.clonebundleprefers=BUNDLESPEC=unknown,BUNDLESPEC=gzip-v2,BUNDLESPEC=bzip2-v2 clone -U http://localhost:$HGPORT prefer-multiple-gz
+ applying clone bundle from http://localhost:$HGPORT1/gz-a.hg
+ adding changesets
+ adding manifests
+ adding file changes
+ added 2 changesets with 2 changes to 2 files
+ finished applying clone bundle
+ searching for changes
+ no changes found
+
+Preferring multiple attributes has correct order
+
+ $ hg --config experimental.clonebundleprefers=extra=b,BUNDLESPEC=bzip2-v2 clone -U http://localhost:$HGPORT prefer-separate-attributes
+ applying clone bundle from http://localhost:$HGPORT1/bz2-b.hg
+ adding changesets
+ adding manifests
+ adding file changes
+ added 2 changesets with 2 changes to 2 files
+ finished applying clone bundle
+ searching for changes
+ no changes found
+
+Test where attribute is missing from some entries
+
+ $ cat > server/.hg/clonebundles.manifest << EOF
+ > http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2
+ > http://localhost:$HGPORT1/bz2-a.hg BUNDLESPEC=bzip2-v2
+ > http://localhost:$HGPORT1/gz-b.hg BUNDLESPEC=gzip-v2 extra=b
+ > http://localhost:$HGPORT1/bz2-b.hg BUNDLESPEC=bzip2-v2 extra=b
+ > EOF
+
+ $ hg --config experimental.clonebundleprefers=extra=b clone -U http://localhost:$HGPORT prefer-partially-defined-attribute
+ applying clone bundle from http://localhost:$HGPORT1/gz-b.hg
+ adding changesets
+ adding manifests
+ adding file changes
+ added 2 changesets with 2 changes to 2 files
+ finished applying clone bundle
+ searching for changes
+ no changes found