annotate mercurial/exchangev2.py @ 40272:a36c5e23c055

rust: iterator bindings to C code In this changeset, still made of Rust code only, we expose the Rust iterator for instantiation and consumption from C code. The idea is that both the index and index_get_parents() will be passed from the C extension, hence avoiding a hard link dependency to parsers.so, so that the crate can still be built and tested independently. On the other hand, parsers.so will use the symbols defined in this changeset.
author Georges Racinet <gracinet@anybox.fr>
date Thu, 27 Sep 2018 16:51:36 +0200
parents b843356d4ae1
children 55836a34f41b
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
39629
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
1 # exchangev2.py - repository exchange for wire protocol version 2
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
2 #
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
3 # Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com>
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
4 #
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
6 # GNU General Public License version 2 or any later version.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
7
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
8 from __future__ import absolute_import
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
9
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
10 import collections
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
11 import weakref
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
12
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
13 from .i18n import _
39629
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
14 from .node import (
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
15 nullid,
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
16 short,
39629
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
17 )
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
18 from . import (
39635
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
19 bookmarks,
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
20 error,
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
21 mdiff,
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
22 phases,
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
23 pycompat,
39629
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
24 setdiscovery,
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
25 )
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
26
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
27 def pull(pullop):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
28 """Pull using wire protocol version 2."""
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
29 repo = pullop.repo
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
30 remote = pullop.remote
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
31 tr = pullop.trmanager.transaction()
39629
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
32
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
33 # Figure out what needs to be fetched.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
34 common, fetch, remoteheads = _pullchangesetdiscovery(
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
35 repo, remote, pullop.heads, abortwhenunrelated=pullop.force)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
36
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
37 # And fetch the data.
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
38 pullheads = pullop.heads or remoteheads
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
39 csetres = _fetchchangesets(repo, tr, remote, common, fetch, pullheads)
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
40
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
41 # New revisions are written to the changelog. But all other updates
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
42 # are deferred. Do those now.
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
43
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
44 # Ensure all new changesets are draft by default. If the repo is
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
45 # publishing, the phase will be adjusted by the loop below.
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
46 if csetres['added']:
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
47 phases.registernew(repo, tr, phases.draft, csetres['added'])
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
48
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
49 # And adjust the phase of all changesets accordingly.
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
50 for phase in phases.phasenames:
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
51 if phase == b'secret' or not csetres['nodesbyphase'][phase]:
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
52 continue
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
53
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
54 phases.advanceboundary(repo, tr, phases.phasenames.index(phase),
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
55 csetres['nodesbyphase'][phase])
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
56
39635
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
57 # Write bookmark updates.
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
58 bookmarks.updatefromremote(repo.ui, repo, csetres['bookmarks'],
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
59 remote.url(), pullop.gettransaction,
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
60 explicit=pullop.explicitbookmarks)
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
61
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
62 manres = _fetchmanifests(repo, tr, remote, csetres['manifestnodes'])
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
63
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
64 # Find all file nodes referenced by added manifests and fetch those
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
65 # revisions.
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
66 fnodes = _derivefilesfrommanifests(repo, manres['added'])
40179
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
67 _fetchfilesfromcsets(repo, tr, remote, fnodes, csetres['added'],
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
68 manres['linkrevs'])
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
69
39629
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
70 def _pullchangesetdiscovery(repo, remote, heads, abortwhenunrelated=True):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
71 """Determine which changesets need to be pulled."""
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
72
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
73 if heads:
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
74 knownnode = repo.changelog.hasnode
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
75 if all(knownnode(head) for head in heads):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
76 return heads, False, heads
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
77
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
78 # TODO wire protocol version 2 is capable of more efficient discovery
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
79 # than setdiscovery. Consider implementing something better.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
80 common, fetch, remoteheads = setdiscovery.findcommonheads(
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
81 repo.ui, repo, remote, abortwhenunrelated=abortwhenunrelated)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
82
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
83 common = set(common)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
84 remoteheads = set(remoteheads)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
85
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
86 # If a remote head is filtered locally, put it back in the common set.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
87 # See the comment in exchange._pulldiscoverychangegroup() for more.
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
88
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
89 if fetch and remoteheads:
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
90 nodemap = repo.unfiltered().changelog.nodemap
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
91
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
92 common |= {head for head in remoteheads if head in nodemap}
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
93
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
94 if set(remoteheads).issubset(common):
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
95 fetch = []
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
96
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
97 common.discard(nullid)
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
98
a86d21e70b2b exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
99 return common, fetch, remoteheads
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
100
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
101 def _fetchchangesets(repo, tr, remote, common, fetch, remoteheads):
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
102 # TODO consider adding a step here where we obtain the DAG shape first
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
103 # (or ask the server to slice changesets into chunks for us) so that
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
104 # we can perform multiple fetches in batches. This will facilitate
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
105 # resuming interrupted clones, higher server-side cache hit rates due
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
106 # to smaller segments, etc.
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
107 with remote.commandexecutor() as e:
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
108 objs = e.callcommand(b'changesetdata', {
40176
41263df08109 wireprotov2: change how revisions are specified to changesetdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40173
diff changeset
109 b'revisions': [{
41263df08109 wireprotov2: change how revisions are specified to changesetdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40173
diff changeset
110 b'type': b'changesetdagrange',
41263df08109 wireprotov2: change how revisions are specified to changesetdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40173
diff changeset
111 b'roots': sorted(common),
41263df08109 wireprotov2: change how revisions are specified to changesetdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40173
diff changeset
112 b'heads': sorted(remoteheads),
41263df08109 wireprotov2: change how revisions are specified to changesetdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40173
diff changeset
113 }],
39635
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
114 b'fields': {b'bookmarks', b'parents', b'phase', b'revision'},
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
115 }).result()
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
116
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
117 # The context manager waits on all response data when exiting. So
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
118 # we need to remain in the context manager in order to stream data.
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
119 return _processchangesetdata(repo, tr, objs)
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
120
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
121 def _processchangesetdata(repo, tr, objs):
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
122 repo.hook('prechangegroup', throw=True,
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
123 **pycompat.strkwargs(tr.hookargs))
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
124
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
125 urepo = repo.unfiltered()
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
126 cl = urepo.changelog
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
127
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
128 cl.delayupdate(tr)
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
129
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
130 # The first emitted object is a header describing the data that
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
131 # follows.
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
132 meta = next(objs)
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
133
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
134 progress = repo.ui.makeprogress(_('changesets'),
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
135 unit=_('chunks'),
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
136 total=meta.get(b'totalitems'))
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
137
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
138 manifestnodes = {}
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
139
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
140 def linkrev(node):
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
141 repo.ui.debug('add changeset %s\n' % short(node))
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
142 # Linkrev for changelog is always self.
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
143 return len(cl)
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
144
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
145 def onchangeset(cl, node):
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
146 progress.increment()
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
147
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
148 revision = cl.changelogrevision(node)
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
149
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
150 # We need to preserve the mapping of changelog revision to node
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
151 # so we can set the linkrev accordingly when manifests are added.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
152 manifestnodes[cl.rev(node)] = revision.manifest
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
153
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
154 nodesbyphase = {phase: set() for phase in phases.phasenames}
39635
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
155 remotebookmarks = {}
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
156
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
157 # addgroup() expects a 7-tuple describing revisions. This normalizes
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
158 # the wire data to that format.
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
159 #
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
160 # This loop also aggregates non-revision metadata, such as phase
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
161 # data.
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
162 def iterrevisions():
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
163 for cset in objs:
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
164 node = cset[b'node']
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
165
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
166 if b'phase' in cset:
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
167 nodesbyphase[cset[b'phase']].add(node)
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
168
39635
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
169 for mark in cset.get(b'bookmarks', []):
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
170 remotebookmarks[mark] = node
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
171
39636
399ddd3227a4 wireprotov2: add TODOs around extending changesetdata fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39635
diff changeset
172 # TODO add mechanism for extensions to examine records so they
399ddd3227a4 wireprotov2: add TODOs around extending changesetdata fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39635
diff changeset
173 # can siphon off custom data fields.
399ddd3227a4 wireprotov2: add TODOs around extending changesetdata fields
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39635
diff changeset
174
39814
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
175 extrafields = {}
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
176
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
177 for field, size in cset.get(b'fieldsfollowing', []):
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
178 extrafields[field] = next(objs)
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
179
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
180 # Some entries might only be metadata only updates.
39814
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
181 if b'revision' not in extrafields:
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
182 continue
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
183
39814
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
184 data = extrafields[b'revision']
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
185
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
186 yield (
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
187 node,
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
188 cset[b'parents'][0],
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
189 cset[b'parents'][1],
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
190 # Linknode is always itself for changesets.
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
191 cset[b'node'],
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
192 # We always send full revisions. So delta base is not set.
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
193 nullid,
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
194 mdiff.trivialdiffheader(len(data)) + data,
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
195 # Flags not yet supported.
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
196 0,
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
197 )
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
198
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
199 added = cl.addgroup(iterrevisions(), linkrev, weakref.proxy(tr),
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
200 addrevisioncb=onchangeset)
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
201
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
202 progress.complete()
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
203
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
204 return {
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
205 'added': added,
39633
ff2de4f2eb3c exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39631
diff changeset
206 'nodesbyphase': nodesbyphase,
39635
349482d726ee exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39633
diff changeset
207 'bookmarks': remotebookmarks,
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
208 'manifestnodes': manifestnodes,
39631
b9e453d683a1 exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39629
diff changeset
209 }
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
210
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
211 def _fetchmanifests(repo, tr, remote, manifestnodes):
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
212 rootmanifest = repo.manifestlog.getstorage(b'')
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
213
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
214 # Some manifests can be shared between changesets. Filter out revisions
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
215 # we already know about.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
216 fetchnodes = []
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
217 linkrevs = {}
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
218 seen = set()
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
219
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
220 for clrev, node in sorted(manifestnodes.iteritems()):
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
221 if node in seen:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
222 continue
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
223
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
224 try:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
225 rootmanifest.rev(node)
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
226 except error.LookupError:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
227 fetchnodes.append(node)
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
228 linkrevs[node] = clrev
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
229
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
230 seen.add(node)
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
231
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
232 # TODO handle tree manifests
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
233
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
234 # addgroup() expects 7-tuple describing revisions. This normalizes
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
235 # the wire data to that format.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
236 def iterrevisions(objs, progress):
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
237 for manifest in objs:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
238 node = manifest[b'node']
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
239
39814
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
240 extrafields = {}
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
241
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
242 for field, size in manifest.get(b'fieldsfollowing', []):
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
243 extrafields[field] = next(objs)
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
244
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
245 if b'delta' in extrafields:
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
246 basenode = manifest[b'deltabasenode']
39814
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
247 delta = extrafields[b'delta']
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
248 elif b'revision' in extrafields:
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
249 basenode = nullid
39814
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
250 revision = extrafields[b'revision']
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
251 delta = mdiff.trivialdiffheader(len(revision)) + revision
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
252 else:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
253 continue
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
254
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
255 yield (
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
256 node,
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
257 manifest[b'parents'][0],
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
258 manifest[b'parents'][1],
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
259 # The value passed in is passed to the lookup function passed
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
260 # to addgroup(). We already have a map of manifest node to
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
261 # changelog revision number. So we just pass in the
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
262 # manifest node here and use linkrevs.__getitem__ as the
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
263 # resolution function.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
264 node,
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
265 basenode,
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
266 delta,
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
267 # Flags not yet supported.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
268 0
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
269 )
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
270
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
271 progress.increment()
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
272
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
273 progress = repo.ui.makeprogress(_('manifests'), unit=_('chunks'),
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
274 total=len(fetchnodes))
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
275
40173
b797150a1ab9 exchangev2: honor server advertised manifestdata recommended batch size
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40035
diff changeset
276 commandmeta = remote.apidescriptor[b'commands'][b'manifestdata']
b797150a1ab9 exchangev2: honor server advertised manifestdata recommended batch size
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40035
diff changeset
277 batchsize = commandmeta.get(b'recommendedbatchsize', 10000)
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
278 # TODO make size configurable on client?
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
279
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
280 # We send commands 1 at a time to the remote. This is not the most
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
281 # efficient because we incur a round trip at the end of each batch.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
282 # However, the existing frame-based reactor keeps consuming server
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
283 # data in the background. And this results in response data buffering
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
284 # in memory. This can consume gigabytes of memory.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
285 # TODO send multiple commands in a request once background buffering
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
286 # issues are resolved.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
287
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
288 added = []
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
289
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
290 for i in pycompat.xrange(0, len(fetchnodes), batchsize):
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
291 batch = [node for node in fetchnodes[i:i + batchsize]]
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
292 if not batch:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
293 continue
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
294
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
295 with remote.commandexecutor() as e:
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
296 objs = e.callcommand(b'manifestdata', {
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
297 b'tree': b'',
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
298 b'nodes': batch,
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
299 b'fields': {b'parents', b'revision'},
39641
aa7e312375cf wireprotov2: let clients drive delta behavior
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39640
diff changeset
300 b'haveparents': True,
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
301 }).result()
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
302
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
303 # Chomp off header object.
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
304 next(objs)
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
305
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
306 added.extend(rootmanifest.addgroup(
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
307 iterrevisions(objs, progress),
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
308 linkrevs.__getitem__,
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
309 weakref.proxy(tr)))
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
310
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
311 progress.complete()
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
312
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
313 return {
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
314 'added': added,
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
315 'linkrevs': linkrevs,
39638
d292328e0143 exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39636
diff changeset
316 }
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
317
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
318 def _derivefilesfrommanifests(repo, manifestnodes):
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
319 """Determine what file nodes are relevant given a set of manifest nodes.
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
320
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
321 Returns a dict mapping file paths to dicts of file node to first manifest
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
322 node.
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
323 """
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
324 ml = repo.manifestlog
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
325 fnodes = collections.defaultdict(dict)
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
326
40035
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
327 progress = repo.ui.makeprogress(
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
328 _('scanning manifests'), total=len(manifestnodes))
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
329
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
330 with progress:
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
331 for manifestnode in manifestnodes:
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
332 m = ml.get(b'', manifestnode)
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
333
40035
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
334 # TODO this will pull in unwanted nodes because it takes the storage
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
335 # delta into consideration. What we really want is something that
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
336 # takes the delta between the manifest's parents. And ideally we
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
337 # would ignore file nodes that are known locally. For now, ignore
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
338 # both these limitations. This will result in incremental fetches
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
339 # requesting data we already have. So this is far from ideal.
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
340 md = m.readfast()
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
341
40035
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
342 for path, fnode in md.items():
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
343 fnodes[path].setdefault(fnode, manifestnode)
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
344
7a347d362a45 exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39814
diff changeset
345 progress.increment()
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
346
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
347 return fnodes
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
348
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
349 def _fetchfiles(repo, tr, remote, fnodes, linkrevs):
40179
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
350 """Fetch file data from explicit file revisions."""
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
351 def iterrevisions(objs, progress):
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
352 for filerevision in objs:
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
353 node = filerevision[b'node']
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
354
39814
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
355 extrafields = {}
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
356
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
357 for field, size in filerevision.get(b'fieldsfollowing', []):
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
358 extrafields[field] = next(objs)
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
359
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
360 if b'delta' in extrafields:
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
361 basenode = filerevision[b'deltabasenode']
39814
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
362 delta = extrafields[b'delta']
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
363 elif b'revision' in extrafields:
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
364 basenode = nullid
39814
d059cb669632 wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39641
diff changeset
365 revision = extrafields[b'revision']
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
366 delta = mdiff.trivialdiffheader(len(revision)) + revision
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
367 else:
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
368 continue
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
369
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
370 yield (
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
371 node,
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
372 filerevision[b'parents'][0],
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
373 filerevision[b'parents'][1],
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
374 node,
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
375 basenode,
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
376 delta,
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
377 # Flags not yet supported.
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
378 0,
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
379 )
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
380
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
381 progress.increment()
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
382
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
383 progress = repo.ui.makeprogress(
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
384 _('files'), unit=_('chunks'),
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
385 total=sum(len(v) for v in fnodes.itervalues()))
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
386
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
387 # TODO make batch size configurable
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
388 batchsize = 10000
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
389 fnodeslist = [x for x in sorted(fnodes.items())]
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
390
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
391 for i in pycompat.xrange(0, len(fnodeslist), batchsize):
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
392 batch = [x for x in fnodeslist[i:i + batchsize]]
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
393 if not batch:
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
394 continue
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
395
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
396 with remote.commandexecutor() as e:
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
397 fs = []
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
398 locallinkrevs = {}
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
399
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
400 for path, nodes in batch:
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
401 fs.append((path, e.callcommand(b'filedata', {
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
402 b'path': path,
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
403 b'nodes': sorted(nodes),
39641
aa7e312375cf wireprotov2: let clients drive delta behavior
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39640
diff changeset
404 b'fields': {b'parents', b'revision'},
aa7e312375cf wireprotov2: let clients drive delta behavior
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39640
diff changeset
405 b'haveparents': True,
39640
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
406 })))
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
407
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
408 locallinkrevs[path] = {
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
409 node: linkrevs[manifestnode]
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
410 for node, manifestnode in nodes.iteritems()}
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
411
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
412 for path, f in fs:
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
413 objs = f.result()
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
414
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
415 # Chomp off header objects.
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
416 next(objs)
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
417
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
418 store = repo.file(path)
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
419 store.addgroup(
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
420 iterrevisions(objs, progress),
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
421 locallinkrevs[path].__getitem__,
039bf1eddc2e exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39638
diff changeset
422 weakref.proxy(tr))
40179
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
423
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
424 def _fetchfilesfromcsets(repo, tr, remote, fnodes, csets, manlinkrevs):
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
425 """Fetch file data from explicit changeset revisions."""
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
426
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
427 def iterrevisions(objs, remaining, progress):
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
428 while remaining:
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
429 filerevision = next(objs)
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
430
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
431 node = filerevision[b'node']
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
432
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
433 extrafields = {}
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
434
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
435 for field, size in filerevision.get(b'fieldsfollowing', []):
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
436 extrafields[field] = next(objs)
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
437
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
438 if b'delta' in extrafields:
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
439 basenode = filerevision[b'deltabasenode']
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
440 delta = extrafields[b'delta']
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
441 elif b'revision' in extrafields:
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
442 basenode = nullid
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
443 revision = extrafields[b'revision']
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
444 delta = mdiff.trivialdiffheader(len(revision)) + revision
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
445 else:
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
446 continue
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
447
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
448 yield (
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
449 node,
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
450 filerevision[b'parents'][0],
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
451 filerevision[b'parents'][1],
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
452 node,
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
453 basenode,
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
454 delta,
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
455 # Flags not yet supported.
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
456 0,
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
457 )
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
458
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
459 progress.increment()
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
460 remaining -= 1
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
461
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
462 progress = repo.ui.makeprogress(
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
463 _('files'), unit=_('chunks'),
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
464 total=sum(len(v) for v in fnodes.itervalues()))
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
465
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
466 commandmeta = remote.apidescriptor[b'commands'][b'filesdata']
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
467 batchsize = commandmeta.get(b'recommendedbatchsize', 50000)
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
468
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
469 for i in pycompat.xrange(0, len(csets), batchsize):
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
470 batch = [x for x in csets[i:i + batchsize]]
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
471 if not batch:
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
472 continue
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
473
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
474 with remote.commandexecutor() as e:
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
475 args = {
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
476 b'revisions': [{
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
477 b'type': b'changesetexplicit',
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
478 b'nodes': batch,
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
479 }],
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
480 b'fields': {b'parents', b'revision'},
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
481 b'haveparents': True,
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
482 }
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
483
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
484 objs = e.callcommand(b'filesdata', args).result()
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
485
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
486 # First object is an overall header.
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
487 overall = next(objs)
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
488
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
489 # We have overall['totalpaths'] segments.
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
490 for i in pycompat.xrange(overall[b'totalpaths']):
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
491 header = next(objs)
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
492
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
493 path = header[b'path']
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
494 store = repo.file(path)
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
495
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
496 linkrevs = {
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
497 fnode: manlinkrevs[mnode]
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
498 for fnode, mnode in fnodes[path].iteritems()}
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
499
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
500 store.addgroup(iterrevisions(objs, header[b'totalitems'],
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
501 progress),
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
502 linkrevs.__getitem__,
b843356d4ae1 exchangev2: use filesdata
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40176
diff changeset
503 weakref.proxy(tr))