annotate mercurial/exchangev2.py @ 39763:5ccd791344f3

localrepo: pass root manifest into manifestlog.__init__ Today, localrepository has a method that can be overloaded which returns an instance of the root manifest storage object. When a manifestlog is created, it calls this private method and stores the root manifest object on it. This "hook" on localrepository isn't part of the documented interface. It isn't compatible with our desire to make repo storage determined before the repo object is constructed. This commit changes manifestlog.__init__ to accept the root storage object instead of calling into the repo to construct it. By doing things this way, the repo instance is responsible for constructing the manifest storage object directly. This does mean that other derived repo types need to overload manifestlog(). But they should have been doing this already, as manifestlog() is typically decorated in a storage-specific way. e.g. localrepository.manifestlog() is decorated as @storecache('00manifest.i'). And this assumes that a 00manifest.i file exists in the store vfs. This condition may not hold for repository types using non-revlog storage. So it is important for special repo types to override manifestlog() to remove this file association. The code changed in perf is wrong because it isn't compatible with older Mercurial versions. But I'm pretty sure the code was broken on older versions before this commit. It only affects `hg perftags`. I don't care enough to fix that at this time. .. api:: ``manifest.manifestlog.__init__()`` now receives the root manifest storage instance instead of calling into a private method on the repo object to obtain it. Differential Revision: https://phab.mercurial-scm.org/D4641
author Gregory Szorc <gregory.szorc@gmail.com>
date Tue, 18 Sep 2018 15:15:24 -0700
parents aa7e312375cf
children d059cb669632
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
39629
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
1 # exchangev2.py - repository exchange for wire protocol version 2
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
2 #
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
3 # Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com>
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
4 #
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
7
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
8 from __future__ import absolute_import
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
9
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
10 import collections
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
11 import weakref
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
12
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
13 from .i18n import _
39629
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
14 from .node import (
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
15 nullid,
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
16 short,
39629
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
17 )
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
18 from . import (
39635
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
19 bookmarks,
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
20 error,
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
21 mdiff,
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
22 phases,
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
23 pycompat,
39629
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
24 setdiscovery,
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
25 )
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
26
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
27 def pull(pullop):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
28 """Pull using wire protocol version 2."""
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
29 repo = pullop.repo
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
30 remote = pullop.remote
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
31 tr = pullop.trmanager.transaction()
39629
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
32
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
33 # Figure out what needs to be fetched.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
34 common, fetch, remoteheads = _pullchangesetdiscovery(
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
35 repo, remote, pullop.heads, abortwhenunrelated=pullop.force)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
36
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
37 # And fetch the data.
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
38 pullheads = pullop.heads or remoteheads
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
39 csetres = _fetchchangesets(repo, tr, remote, common, fetch, pullheads)
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
40
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
41 # New revisions are written to the changelog. But all other updates
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
42 # are deferred. Do those now.
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
43
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
44 # Ensure all new changesets are draft by default. If the repo is
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
45 # publishing, the phase will be adjusted by the loop below.
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
46 if csetres['added']:
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
47 phases.registernew(repo, tr, phases.draft, csetres['added'])
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
48
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
49 # And adjust the phase of all changesets accordingly.
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
50 for phase in phases.phasenames:
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
51 if phase == b'secret' or not csetres['nodesbyphase'][phase]:
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
52 continue
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
53
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
54 phases.advanceboundary(repo, tr, phases.phasenames.index(phase),
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
55 csetres['nodesbyphase'][phase])
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
56
39635
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
57 # Write bookmark updates.
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
58 bookmarks.updatefromremote(repo.ui, repo, csetres['bookmarks'],
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
59 remote.url(), pullop.gettransaction,
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
60 explicit=pullop.explicitbookmarks)
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
61
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
62 manres = _fetchmanifests(repo, tr, remote, csetres['manifestnodes'])
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
63
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
64 # Find all file nodes referenced by added manifests and fetch those
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
65 # revisions.
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
66 fnodes = _derivefilesfrommanifests(repo, manres['added'])
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
67 _fetchfiles(repo, tr, remote, fnodes, manres['linkrevs'])
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
68
39629
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
69 def _pullchangesetdiscovery(repo, remote, heads, abortwhenunrelated=True):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
70 """Determine which changesets need to be pulled."""
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
71
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
72 if heads:
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
73 knownnode = repo.changelog.hasnode
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
74 if all(knownnode(head) for head in heads):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
75 return heads, False, heads
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
76
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
77 # TODO wire protocol version 2 is capable of more efficient discovery
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
78 # than setdiscovery. Consider implementing something better.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
79 common, fetch, remoteheads = setdiscovery.findcommonheads(
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
80 repo.ui, repo, remote, abortwhenunrelated=abortwhenunrelated)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
81
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
82 common = set(common)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
83 remoteheads = set(remoteheads)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
84
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
85 # If a remote head is filtered locally, put it back in the common set.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
86 # See the comment in exchange._pulldiscoverychangegroup() for more.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
87
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
88 if fetch and remoteheads:
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
89 nodemap = repo.unfiltered().changelog.nodemap
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
90
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
91 common |= {head for head in remoteheads if head in nodemap}
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
92
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
93 if set(remoteheads).issubset(common):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
94 fetch = []
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
95
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
96 common.discard(nullid)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
97
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
98 return common, fetch, remoteheads
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
99
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
100 def _fetchchangesets(repo, tr, remote, common, fetch, remoteheads):
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
101 # TODO consider adding a step here where we obtain the DAG shape first
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
102 # (or ask the server to slice changesets into chunks for us) so that
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
103 # we can perform multiple fetches in batches. This will facilitate
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
104 # resuming interrupted clones, higher server-side cache hit rates due
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
105 # to smaller segments, etc.
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
106 with remote.commandexecutor() as e:
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
107 objs = e.callcommand(b'changesetdata', {
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
108 b'noderange': [sorted(common), sorted(remoteheads)],
39635
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
109 b'fields': {b'bookmarks', b'parents', b'phase', b'revision'},
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
110 }).result()
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
111
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
112 # The context manager waits on all response data when exiting. So
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
113 # we need to remain in the context manager in order to stream data.
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
114 return _processchangesetdata(repo, tr, objs)
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
115
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
116 def _processchangesetdata(repo, tr, objs):
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
117 repo.hook('prechangegroup', throw=True,
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
118 **pycompat.strkwargs(tr.hookargs))
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
119
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
120 urepo = repo.unfiltered()
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
121 cl = urepo.changelog
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
122
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
123 cl.delayupdate(tr)
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
124
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
125 # The first emitted object is a header describing the data that
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
126 # follows.
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
127 meta = next(objs)
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
128
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
129 progress = repo.ui.makeprogress(_('changesets'),
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
130 unit=_('chunks'),
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
131 total=meta.get(b'totalitems'))
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
132
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
133 manifestnodes = {}
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
134
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
135 def linkrev(node):
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
136 repo.ui.debug('add changeset %s\n' % short(node))
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
137 # Linkrev for changelog is always self.
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
138 return len(cl)
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
139
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
140 def onchangeset(cl, node):
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
141 progress.increment()
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
142
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
143 revision = cl.changelogrevision(node)
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
144
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
145 # We need to preserve the mapping of changelog revision to node
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
146 # so we can set the linkrev accordingly when manifests are added.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
147 manifestnodes[cl.rev(node)] = revision.manifest
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
148
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
149 nodesbyphase = {phase: set() for phase in phases.phasenames}
39635
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
150 remotebookmarks = {}
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
151
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
152 # addgroup() expects a 7-tuple describing revisions. This normalizes
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
153 # the wire data to that format.
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
154 #
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
155 # This loop also aggregates non-revision metadata, such as phase
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
156 # data.
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
157 def iterrevisions():
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
158 for cset in objs:
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
159 node = cset[b'node']
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
160
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
161 if b'phase' in cset:
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
162 nodesbyphase[cset[b'phase']].add(node)
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
163
39635
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
164 for mark in cset.get(b'bookmarks', []):
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
165 remotebookmarks[mark] = node
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
166
39636
399ddd3227a4 wireprotov2: add TODOs around extending changesetdata fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39635
diff changeset
167 # TODO add mechanism for extensions to examine records so they
399ddd3227a4 wireprotov2: add TODOs around extending changesetdata fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39635
diff changeset
168 # can siphon off custom data fields.
399ddd3227a4 wireprotov2: add TODOs around extending changesetdata fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39635
diff changeset
169
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
170 # Some entries might only be metadata only updates.
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
171 if b'revisionsize' not in cset:
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
172 continue
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
173
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
174 data = next(objs)
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
175
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
176 yield (
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
177 node,
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
178 cset[b'parents'][0],
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
179 cset[b'parents'][1],
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
180 # Linknode is always itself for changesets.
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
181 cset[b'node'],
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
182 # We always send full revisions. So delta base is not set.
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
183 nullid,
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
184 mdiff.trivialdiffheader(len(data)) + data,
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
185 # Flags not yet supported.
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
186 0,
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
187 )
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
188
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
189 added = cl.addgroup(iterrevisions(), linkrev, weakref.proxy(tr),
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
190 addrevisioncb=onchangeset)
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
191
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
192 progress.complete()
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
193
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
194 return {
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
195 'added': added,
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
196 'nodesbyphase': nodesbyphase,
39635
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
197 'bookmarks': remotebookmarks,
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
198 'manifestnodes': manifestnodes,
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
199 }
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
200
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
201 def _fetchmanifests(repo, tr, remote, manifestnodes):
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
202 rootmanifest = repo.manifestlog.getstorage(b'')
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
203
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
204 # Some manifests can be shared between changesets. Filter out revisions
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
205 # we already know about.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
206 fetchnodes = []
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
207 linkrevs = {}
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
208 seen = set()
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
209
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
210 for clrev, node in sorted(manifestnodes.iteritems()):
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
211 if node in seen:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
212 continue
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
213
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
214 try:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
215 rootmanifest.rev(node)
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
216 except error.LookupError:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
217 fetchnodes.append(node)
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
218 linkrevs[node] = clrev
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
219
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
220 seen.add(node)
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
221
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
222 # TODO handle tree manifests
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
223
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
224 # addgroup() expects 7-tuple describing revisions. This normalizes
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
225 # the wire data to that format.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
226 def iterrevisions(objs, progress):
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
227 for manifest in objs:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
228 node = manifest[b'node']
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
229
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
230 if b'deltasize' in manifest:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
231 basenode = manifest[b'deltabasenode']
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
232 delta = next(objs)
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
233 elif b'revisionsize' in manifest:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
234 basenode = nullid
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
235 revision = next(objs)
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
236 delta = mdiff.trivialdiffheader(len(revision)) + revision
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
237 else:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
238 continue
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
239
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
240 yield (
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
241 node,
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
242 manifest[b'parents'][0],
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
243 manifest[b'parents'][1],
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
244 # The value passed in is passed to the lookup function passed
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
245 # to addgroup(). We already have a map of manifest node to
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
246 # changelog revision number. So we just pass in the
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
247 # manifest node here and use linkrevs.__getitem__ as the
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
248 # resolution function.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
249 node,
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
250 basenode,
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
251 delta,
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
252 # Flags not yet supported.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
253 0
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
254 )
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
255
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
256 progress.increment()
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
257
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
258 progress = repo.ui.makeprogress(_('manifests'), unit=_('chunks'),
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
259 total=len(fetchnodes))
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
260
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
261 # Fetch manifests 10,000 per command.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
262 # TODO have server advertise preferences?
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
263 # TODO make size configurable on client?
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
264 batchsize = 10000
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
265
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
266 # We send commands 1 at a time to the remote. This is not the most
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
267 # efficient because we incur a round trip at the end of each batch.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
268 # However, the existing frame-based reactor keeps consuming server
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
269 # data in the background. And this results in response data buffering
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
270 # in memory. This can consume gigabytes of memory.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
271 # TODO send multiple commands in a request once background buffering
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
272 # issues are resolved.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
273
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
274 added = []
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
275
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
276 for i in pycompat.xrange(0, len(fetchnodes), batchsize):
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
277 batch = [node for node in fetchnodes[i:i + batchsize]]
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
278 if not batch:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
279 continue
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
280
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
281 with remote.commandexecutor() as e:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
282 objs = e.callcommand(b'manifestdata', {
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
283 b'tree': b'',
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
284 b'nodes': batch,
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
285 b'fields': {b'parents', b'revision'},
39641
aa7e312375cf wireprotov2: let clients drive delta behavior
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39640
diff changeset
286 b'haveparents': True,
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
287 }).result()
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
288
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
289 # Chomp off header object.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
290 next(objs)
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
291
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
292 added.extend(rootmanifest.addgroup(
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
293 iterrevisions(objs, progress),
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
294 linkrevs.__getitem__,
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
295 weakref.proxy(tr)))
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
296
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
297 progress.complete()
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
298
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
299 return {
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
300 'added': added,
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
301 'linkrevs': linkrevs,
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
302 }
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
303
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
304 def _derivefilesfrommanifests(repo, manifestnodes):
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
305 """Determine what file nodes are relevant given a set of manifest nodes.
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
306
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
307 Returns a dict mapping file paths to dicts of file node to first manifest
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
308 node.
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
309 """
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
310 ml = repo.manifestlog
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
311 fnodes = collections.defaultdict(dict)
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
312
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
313 for manifestnode in manifestnodes:
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
314 m = ml.get(b'', manifestnode)
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
315
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
316 # TODO this will pull in unwanted nodes because it takes the storage
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
317 # delta into consideration. What we really want is something that takes
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
318 # the delta between the manifest's parents. And ideally we would
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
319 # ignore file nodes that are known locally. For now, ignore both
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
320 # these limitations. This will result in incremental fetches requesting
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
321 # data we already have. So this is far from ideal.
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
322 md = m.readfast()
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
323
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
324 for path, fnode in md.items():
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
325 fnodes[path].setdefault(fnode, manifestnode)
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
326
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
327 return fnodes
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
328
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
329 def _fetchfiles(repo, tr, remote, fnodes, linkrevs):
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
330 def iterrevisions(objs, progress):
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
331 for filerevision in objs:
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
332 node = filerevision[b'node']
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
333
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
334 if b'deltasize' in filerevision:
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
335 basenode = filerevision[b'deltabasenode']
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
336 delta = next(objs)
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
337 elif b'revisionsize' in filerevision:
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
338 basenode = nullid
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
339 revision = next(objs)
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
340 delta = mdiff.trivialdiffheader(len(revision)) + revision
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
341 else:
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
342 continue
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
343
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
344 yield (
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
345 node,
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
346 filerevision[b'parents'][0],
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
347 filerevision[b'parents'][1],
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
348 node,
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
349 basenode,
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
350 delta,
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
351 # Flags not yet supported.
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
352 0,
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
353 )
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
354
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
355 progress.increment()
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
356
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
357 progress = repo.ui.makeprogress(
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
358 _('files'), unit=_('chunks'),
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
359 total=sum(len(v) for v in fnodes.itervalues()))
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
360
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
361 # TODO make batch size configurable
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
362 batchsize = 10000
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
363 fnodeslist = [x for x in sorted(fnodes.items())]
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
364
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
365 for i in pycompat.xrange(0, len(fnodeslist), batchsize):
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
366 batch = [x for x in fnodeslist[i:i + batchsize]]
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
367 if not batch:
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
368 continue
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
369
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
370 with remote.commandexecutor() as e:
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
371 fs = []
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
372 locallinkrevs = {}
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
373
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
374 for path, nodes in batch:
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
375 fs.append((path, e.callcommand(b'filedata', {
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
376 b'path': path,
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
377 b'nodes': sorted(nodes),
39641
aa7e312375cf wireprotov2: let clients drive delta behavior
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39640
diff changeset
378 b'fields': {b'parents', b'revision'},
aa7e312375cf wireprotov2: let clients drive delta behavior
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39640
diff changeset
379 b'haveparents': True,
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
380 })))
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
381
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
382 locallinkrevs[path] = {
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
383 node: linkrevs[manifestnode]
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
384 for node, manifestnode in nodes.iteritems()}
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
385
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
386 for path, f in fs:
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
387 objs = f.result()
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
388
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
389 # Chomp off header objects.
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
390 next(objs)
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
391
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
392 store = repo.file(path)
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
393 store.addgroup(
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
394 iterrevisions(objs, progress),
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
395 locallinkrevs[path].__getitem__,
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
396 weakref.proxy(tr))