Mercurial > hg
annotate mercurial/exchangev2.py @ 40176:41263df08109
wireprotov2: change how revisions are specified to changesetdata
Right now, we have a handful of arguments for specifying the revisions
whose data should be returned. Defining how all these arguments
interact when various combinations are present is difficult.
This commit establishes a new, generic mechanism for specifying
revisions. Instead of a hodgepodge of arguments defining things, we
have a list of dicts that specify revision selectors. The final set
of revisions is a union of all these selectors.
We implement support for specifying revisions based on:
* An explicit list of changeset revisions
* An explicit list of changeset revisions plus ancestry depth
* A DAG range between changeset roots and heads
If you squint hard enough, this problem has already been solved by
revsets. But I'm reluctant to expose revsets to the wire protocol
because that would require servers to implement a revset parser.
Plus there are security and performance implications: the set
of revision selectors needs to be narrowly and specifically tailored
for what is appropriate to be executing on a server. Perhaps there
would be a way for us to express the "parse tree" of a revset
query, for example. I'm not sure. We can explore this space another
time. For now, the new mechanism should bring sufficient flexibility
while remaining relatively simple.
The selector "types" are prefixed with "changeset" because I plan
to add manifest and file-flavored selectors as well. This will enable
us to e.g. select file revisions based on a range of changeset
revisions.
Differential Revision: https://phab.mercurial-scm.org/D4979
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Mon, 08 Oct 2018 18:17:12 -0700 |
parents | b797150a1ab9 |
children | b843356d4ae1 |
rev | line source |
---|---|
39629
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
1 # exchangev2.py - repository exchange for wire protocol version 2 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
2 # |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
3 # Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com> |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
4 # |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
5 # This software may be used and distributed according to the terms of the |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
6 # GNU General Public License version 2 or any later version. |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
7 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
8 from __future__ import absolute_import |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
9 |
39640
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
10 import collections |
39631
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
11 import weakref |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
12 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
13 from .i18n import _ |
39629
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
14 from .node import ( |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
15 nullid, |
39631
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
16 short, |
39629
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
17 ) |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
18 from . import ( |
39635
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39633
diff
changeset
|
19 bookmarks, |
39638
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
20 error, |
39631
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
21 mdiff, |
39633
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
22 phases, |
39631
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
23 pycompat, |
39629
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
24 setdiscovery, |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
25 ) |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
26 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
27 def pull(pullop): |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
28 """Pull using wire protocol version 2.""" |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
29 repo = pullop.repo |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
30 remote = pullop.remote |
39631
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
31 tr = pullop.trmanager.transaction() |
39629
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
32 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
33 # Figure out what needs to be fetched. |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
34 common, fetch, remoteheads = _pullchangesetdiscovery( |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
35 repo, remote, pullop.heads, abortwhenunrelated=pullop.force) |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
36 |
39633
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
37 # And fetch the data. |
39631
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
38 pullheads = pullop.heads or remoteheads |
39633
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
39 csetres = _fetchchangesets(repo, tr, remote, common, fetch, pullheads) |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
40 |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
41 # New revisions are written to the changelog. But all other updates |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
42 # are deferred. Do those now. |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
43 |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
44 # Ensure all new changesets are draft by default. If the repo is |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
45 # publishing, the phase will be adjusted by the loop below. |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
46 if csetres['added']: |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
47 phases.registernew(repo, tr, phases.draft, csetres['added']) |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
48 |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
49 # And adjust the phase of all changesets accordingly. |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
50 for phase in phases.phasenames: |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
51 if phase == b'secret' or not csetres['nodesbyphase'][phase]: |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
52 continue |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
53 |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
54 phases.advanceboundary(repo, tr, phases.phasenames.index(phase), |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
55 csetres['nodesbyphase'][phase]) |
39631
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
56 |
39635
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39633
diff
changeset
|
57 # Write bookmark updates. |
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39633
diff
changeset
|
58 bookmarks.updatefromremote(repo.ui, repo, csetres['bookmarks'], |
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39633
diff
changeset
|
59 remote.url(), pullop.gettransaction, |
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39633
diff
changeset
|
60 explicit=pullop.explicitbookmarks) |
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39633
diff
changeset
|
61 |
39640
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
62 manres = _fetchmanifests(repo, tr, remote, csetres['manifestnodes']) |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
63 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
64 # Find all file nodes referenced by added manifests and fetch those |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
65 # revisions. |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
66 fnodes = _derivefilesfrommanifests(repo, manres['added']) |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
67 _fetchfiles(repo, tr, remote, fnodes, manres['linkrevs']) |
39638
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
68 |
39629
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
69 def _pullchangesetdiscovery(repo, remote, heads, abortwhenunrelated=True): |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
70 """Determine which changesets need to be pulled.""" |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
71 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
72 if heads: |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
73 knownnode = repo.changelog.hasnode |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
74 if all(knownnode(head) for head in heads): |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
75 return heads, False, heads |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
76 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
77 # TODO wire protocol version 2 is capable of more efficient discovery |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
78 # than setdiscovery. Consider implementing something better. |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
79 common, fetch, remoteheads = setdiscovery.findcommonheads( |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
80 repo.ui, repo, remote, abortwhenunrelated=abortwhenunrelated) |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
81 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
82 common = set(common) |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
83 remoteheads = set(remoteheads) |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
84 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
85 # If a remote head is filtered locally, put it back in the common set. |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
86 # See the comment in exchange._pulldiscoverychangegroup() for more. |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
87 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
88 if fetch and remoteheads: |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
89 nodemap = repo.unfiltered().changelog.nodemap |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
90 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
91 common |= {head for head in remoteheads if head in nodemap} |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
92 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
93 if set(remoteheads).issubset(common): |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
94 fetch = [] |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
95 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
96 common.discard(nullid) |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
97 |
a86d21e70b2b
exchangev2: start to implement pull with wire protocol v2
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff
changeset
|
98 return common, fetch, remoteheads |
39631
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
99 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
100 def _fetchchangesets(repo, tr, remote, common, fetch, remoteheads): |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
101 # TODO consider adding a step here where we obtain the DAG shape first |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
102 # (or ask the server to slice changesets into chunks for us) so that |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
103 # we can perform multiple fetches in batches. This will facilitate |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
104 # resuming interrupted clones, higher server-side cache hit rates due |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
105 # to smaller segments, etc. |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
106 with remote.commandexecutor() as e: |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
107 objs = e.callcommand(b'changesetdata', { |
40176
41263df08109
wireprotov2: change how revisions are specified to changesetdata
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40173
diff
changeset
|
108 b'revisions': [{ |
41263df08109
wireprotov2: change how revisions are specified to changesetdata
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40173
diff
changeset
|
109 b'type': b'changesetdagrange', |
41263df08109
wireprotov2: change how revisions are specified to changesetdata
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40173
diff
changeset
|
110 b'roots': sorted(common), |
41263df08109
wireprotov2: change how revisions are specified to changesetdata
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40173
diff
changeset
|
111 b'heads': sorted(remoteheads), |
41263df08109
wireprotov2: change how revisions are specified to changesetdata
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40173
diff
changeset
|
112 }], |
39635
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39633
diff
changeset
|
113 b'fields': {b'bookmarks', b'parents', b'phase', b'revision'}, |
39631
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
114 }).result() |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
115 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
116 # The context manager waits on all response data when exiting. So |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
117 # we need to remain in the context manager in order to stream data. |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
118 return _processchangesetdata(repo, tr, objs) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
119 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
120 def _processchangesetdata(repo, tr, objs): |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
121 repo.hook('prechangegroup', throw=True, |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
122 **pycompat.strkwargs(tr.hookargs)) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
123 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
124 urepo = repo.unfiltered() |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
125 cl = urepo.changelog |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
126 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
127 cl.delayupdate(tr) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
128 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
129 # The first emitted object is a header describing the data that |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
130 # follows. |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
131 meta = next(objs) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
132 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
133 progress = repo.ui.makeprogress(_('changesets'), |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
134 unit=_('chunks'), |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
135 total=meta.get(b'totalitems')) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
136 |
39638
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
137 manifestnodes = {} |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
138 |
39631
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
139 def linkrev(node): |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
140 repo.ui.debug('add changeset %s\n' % short(node)) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
141 # Linkrev for changelog is always self. |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
142 return len(cl) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
143 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
144 def onchangeset(cl, node): |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
145 progress.increment() |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
146 |
39638
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
147 revision = cl.changelogrevision(node) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
148 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
149 # We need to preserve the mapping of changelog revision to node |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
150 # so we can set the linkrev accordingly when manifests are added. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
151 manifestnodes[cl.rev(node)] = revision.manifest |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
152 |
39633
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
153 nodesbyphase = {phase: set() for phase in phases.phasenames} |
39635
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39633
diff
changeset
|
154 remotebookmarks = {} |
39633
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
155 |
39631
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
156 # addgroup() expects a 7-tuple describing revisions. This normalizes |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
157 # the wire data to that format. |
39633
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
158 # |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
159 # This loop also aggregates non-revision metadata, such as phase |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
160 # data. |
39631
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
161 def iterrevisions(): |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
162 for cset in objs: |
39633
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
163 node = cset[b'node'] |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
164 |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
165 if b'phase' in cset: |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
166 nodesbyphase[cset[b'phase']].add(node) |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
167 |
39635
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39633
diff
changeset
|
168 for mark in cset.get(b'bookmarks', []): |
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39633
diff
changeset
|
169 remotebookmarks[mark] = node |
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39633
diff
changeset
|
170 |
39636
399ddd3227a4
wireprotov2: add TODOs around extending changesetdata fields
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39635
diff
changeset
|
171 # TODO add mechanism for extensions to examine records so they |
399ddd3227a4
wireprotov2: add TODOs around extending changesetdata fields
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39635
diff
changeset
|
172 # can siphon off custom data fields. |
399ddd3227a4
wireprotov2: add TODOs around extending changesetdata fields
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39635
diff
changeset
|
173 |
39814
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
174 extrafields = {} |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
175 |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
176 for field, size in cset.get(b'fieldsfollowing', []): |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
177 extrafields[field] = next(objs) |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
178 |
39633
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
179 # Some entries might only be metadata only updates. |
39814
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
180 if b'revision' not in extrafields: |
39633
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
181 continue |
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
182 |
39814
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
183 data = extrafields[b'revision'] |
39631
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
184 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
185 yield ( |
39633
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
186 node, |
39631
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
187 cset[b'parents'][0], |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
188 cset[b'parents'][1], |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
189 # Linknode is always itself for changesets. |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
190 cset[b'node'], |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
191 # We always send full revisions. So delta base is not set. |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
192 nullid, |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
193 mdiff.trivialdiffheader(len(data)) + data, |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
194 # Flags not yet supported. |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
195 0, |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
196 ) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
197 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
198 added = cl.addgroup(iterrevisions(), linkrev, weakref.proxy(tr), |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
199 addrevisioncb=onchangeset) |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
200 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
201 progress.complete() |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
202 |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
203 return { |
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
204 'added': added, |
39633
ff2de4f2eb3c
exchangev2: fetch and apply phases data
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39631
diff
changeset
|
205 'nodesbyphase': nodesbyphase, |
39635
349482d726ee
exchangev2: fetch and apply bookmarks
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39633
diff
changeset
|
206 'bookmarks': remotebookmarks, |
39638
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
207 'manifestnodes': manifestnodes, |
39631
b9e453d683a1
exchangev2: fetch changeset revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39629
diff
changeset
|
208 } |
39638
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
209 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
210 def _fetchmanifests(repo, tr, remote, manifestnodes): |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
211 rootmanifest = repo.manifestlog.getstorage(b'') |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
212 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
213 # Some manifests can be shared between changesets. Filter out revisions |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
214 # we already know about. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
215 fetchnodes = [] |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
216 linkrevs = {} |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
217 seen = set() |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
218 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
219 for clrev, node in sorted(manifestnodes.iteritems()): |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
220 if node in seen: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
221 continue |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
222 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
223 try: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
224 rootmanifest.rev(node) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
225 except error.LookupError: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
226 fetchnodes.append(node) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
227 linkrevs[node] = clrev |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
228 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
229 seen.add(node) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
230 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
231 # TODO handle tree manifests |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
232 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
233 # addgroup() expects 7-tuple describing revisions. This normalizes |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
234 # the wire data to that format. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
235 def iterrevisions(objs, progress): |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
236 for manifest in objs: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
237 node = manifest[b'node'] |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
238 |
39814
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
239 extrafields = {} |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
240 |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
241 for field, size in manifest.get(b'fieldsfollowing', []): |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
242 extrafields[field] = next(objs) |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
243 |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
244 if b'delta' in extrafields: |
39638
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
245 basenode = manifest[b'deltabasenode'] |
39814
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
246 delta = extrafields[b'delta'] |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
247 elif b'revision' in extrafields: |
39638
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
248 basenode = nullid |
39814
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
249 revision = extrafields[b'revision'] |
39638
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
250 delta = mdiff.trivialdiffheader(len(revision)) + revision |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
251 else: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
252 continue |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
253 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
254 yield ( |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
255 node, |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
256 manifest[b'parents'][0], |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
257 manifest[b'parents'][1], |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
258 # The value passed in is passed to the lookup function passed |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
259 # to addgroup(). We already have a map of manifest node to |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
260 # changelog revision number. So we just pass in the |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
261 # manifest node here and use linkrevs.__getitem__ as the |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
262 # resolution function. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
263 node, |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
264 basenode, |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
265 delta, |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
266 # Flags not yet supported. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
267 0 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
268 ) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
269 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
270 progress.increment() |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
271 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
272 progress = repo.ui.makeprogress(_('manifests'), unit=_('chunks'), |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
273 total=len(fetchnodes)) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
274 |
40173
b797150a1ab9
exchangev2: honor server advertised manifestdata recommended batch size
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40035
diff
changeset
|
275 commandmeta = remote.apidescriptor[b'commands'][b'manifestdata'] |
b797150a1ab9
exchangev2: honor server advertised manifestdata recommended batch size
Gregory Szorc <gregory.szorc@gmail.com>
parents:
40035
diff
changeset
|
276 batchsize = commandmeta.get(b'recommendedbatchsize', 10000) |
39638
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
277 # TODO make size configurable on client? |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
278 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
279 # We send commands 1 at a time to the remote. This is not the most |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
280 # efficient because we incur a round trip at the end of each batch. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
281 # However, the existing frame-based reactor keeps consuming server |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
282 # data in the background. And this results in response data buffering |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
283 # in memory. This can consume gigabytes of memory. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
284 # TODO send multiple commands in a request once background buffering |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
285 # issues are resolved. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
286 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
287 added = [] |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
288 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
289 for i in pycompat.xrange(0, len(fetchnodes), batchsize): |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
290 batch = [node for node in fetchnodes[i:i + batchsize]] |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
291 if not batch: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
292 continue |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
293 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
294 with remote.commandexecutor() as e: |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
295 objs = e.callcommand(b'manifestdata', { |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
296 b'tree': b'', |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
297 b'nodes': batch, |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
298 b'fields': {b'parents', b'revision'}, |
39641
aa7e312375cf
wireprotov2: let clients drive delta behavior
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39640
diff
changeset
|
299 b'haveparents': True, |
39638
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
300 }).result() |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
301 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
302 # Chomp off header object. |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
303 next(objs) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
304 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
305 added.extend(rootmanifest.addgroup( |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
306 iterrevisions(objs, progress), |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
307 linkrevs.__getitem__, |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
308 weakref.proxy(tr))) |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
309 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
310 progress.complete() |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
311 |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
312 return { |
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
313 'added': added, |
39640
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
314 'linkrevs': linkrevs, |
39638
d292328e0143
exchangev2: fetch manifest revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39636
diff
changeset
|
315 } |
39640
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
316 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
317 def _derivefilesfrommanifests(repo, manifestnodes): |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
318 """Determine what file nodes are relevant given a set of manifest nodes. |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
319 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
320 Returns a dict mapping file paths to dicts of file node to first manifest |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
321 node. |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
322 """ |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
323 ml = repo.manifestlog |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
324 fnodes = collections.defaultdict(dict) |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
325 |
40035
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
326 progress = repo.ui.makeprogress( |
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
327 _('scanning manifests'), total=len(manifestnodes)) |
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
328 |
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
329 with progress: |
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
330 for manifestnode in manifestnodes: |
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
331 m = ml.get(b'', manifestnode) |
39640
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
332 |
40035
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
333 # TODO this will pull in unwanted nodes because it takes the storage |
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
334 # delta into consideration. What we really want is something that |
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
335 # takes the delta between the manifest's parents. And ideally we |
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
336 # would ignore file nodes that are known locally. For now, ignore |
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
337 # both these limitations. This will result in incremental fetches |
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
338 # requesting data we already have. So this is far from ideal. |
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
339 md = m.readfast() |
39640
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
340 |
40035
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
341 for path, fnode in md.items(): |
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
342 fnodes[path].setdefault(fnode, manifestnode) |
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
343 |
7a347d362a45
exchangev2: add progress bar around manifest scanning
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39814
diff
changeset
|
344 progress.increment() |
39640
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
345 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
346 return fnodes |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
347 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
348 def _fetchfiles(repo, tr, remote, fnodes, linkrevs): |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
349 def iterrevisions(objs, progress): |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
350 for filerevision in objs: |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
351 node = filerevision[b'node'] |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
352 |
39814
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
353 extrafields = {} |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
354 |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
355 for field, size in filerevision.get(b'fieldsfollowing', []): |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
356 extrafields[field] = next(objs) |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
357 |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
358 if b'delta' in extrafields: |
39640
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
359 basenode = filerevision[b'deltabasenode'] |
39814
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
360 delta = extrafields[b'delta'] |
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
361 elif b'revision' in extrafields: |
39640
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
362 basenode = nullid |
39814
d059cb669632
wireprotov2: allow multiple fields to follow revision maps
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39641
diff
changeset
|
363 revision = extrafields[b'revision'] |
39640
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
364 delta = mdiff.trivialdiffheader(len(revision)) + revision |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
365 else: |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
366 continue |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
367 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
368 yield ( |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
369 node, |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
370 filerevision[b'parents'][0], |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
371 filerevision[b'parents'][1], |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
372 node, |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
373 basenode, |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
374 delta, |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
375 # Flags not yet supported. |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
376 0, |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
377 ) |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
378 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
379 progress.increment() |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
380 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
381 progress = repo.ui.makeprogress( |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
382 _('files'), unit=_('chunks'), |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
383 total=sum(len(v) for v in fnodes.itervalues())) |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
384 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
385 # TODO make batch size configurable |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
386 batchsize = 10000 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
387 fnodeslist = [x for x in sorted(fnodes.items())] |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
388 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
389 for i in pycompat.xrange(0, len(fnodeslist), batchsize): |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
390 batch = [x for x in fnodeslist[i:i + batchsize]] |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
391 if not batch: |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
392 continue |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
393 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
394 with remote.commandexecutor() as e: |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
395 fs = [] |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
396 locallinkrevs = {} |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
397 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
398 for path, nodes in batch: |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
399 fs.append((path, e.callcommand(b'filedata', { |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
400 b'path': path, |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
401 b'nodes': sorted(nodes), |
39641
aa7e312375cf
wireprotov2: let clients drive delta behavior
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39640
diff
changeset
|
402 b'fields': {b'parents', b'revision'}, |
aa7e312375cf
wireprotov2: let clients drive delta behavior
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39640
diff
changeset
|
403 b'haveparents': True, |
39640
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
404 }))) |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
405 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
406 locallinkrevs[path] = { |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
407 node: linkrevs[manifestnode] |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
408 for node, manifestnode in nodes.iteritems()} |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
409 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
410 for path, f in fs: |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
411 objs = f.result() |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
412 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
413 # Chomp off header objects. |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
414 next(objs) |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
415 |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
416 store = repo.file(path) |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
417 store.addgroup( |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
418 iterrevisions(objs, progress), |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
419 locallinkrevs[path].__getitem__, |
039bf1eddc2e
exchangev2: fetch file revisions
Gregory Szorc <gregory.szorc@gmail.com>
parents:
39638
diff
changeset
|
420 weakref.proxy(tr)) |