thankyou! i really appreciate it!
2016-04-30 4:29 GMT+08:00 Josh Wills <josh.wills@gmail.com>:
> Yikes, what idiot wrote this code? ;-)
>
> So I'll be honest, I couldn't figure out what the code was doing either--
> so I went back in time (to 2012!) using git blame to figure out when/why I
> added it, and discovered that it was a special case to handle map-side
> outputs that had a downstream GBK operation, via this commit:
>
>
> https://github.com/apache/crunch/commit/28e51b6a4505ff406c0d9472303c28cd2e2d6aaa
>
> After staring at it for awhile, I *think* the reason this works is because
> this line:
>
> graph.getEdge(vertex, splitTail).addNodePath(headPath);
>
> doesn't actually do anything-- the headPath here is always empty, so
> there's no impact to the final graph. I'd be curious if anything failed if
> we removed that line.
>
> Josh
>
> On Wed, Apr 27, 2016 at 11:17 PM, 陈竞 <cj.magina@gmail.com> wrote:
>
>> i'm reading crunch source code, i am very confused about the code
>> of MSCRPlanner.prepareFinalGraph():
>>
>> if (baseVertex.isGBK()) {
>> Vertex vertex = graph.getVertexAt(baseVertex.getPCollection());
>> for (Edge e : baseVertex.getIncomingEdges()) {
>> if (e.getHead().isOutput()) {
>> // Execute an edge split.
>> Vertex splitTail = e.getHead();
>> PCollectionImpl<?> split = splitTail.getPCollection();
>> InputCollection<?> inputNode = handleSplitTarget(split);
>> Vertex splitHead = graph.addVertex(inputNode, false);
>>
>> // Divide up the node paths in the edge between the two GBK nodes so
>> // that each node is either owned by GBK1 -> newTail or newHead -> GBK2.
>> for (NodePath path : e.getNodePaths()) {
>> NodePath headPath = path.splitAt(split, splitHead.getPCollection());
>> graph.getEdge(vertex, splitTail).addNodePath(headPath);
>> graph.getEdge(splitHead, vertex).addNodePath(path);
>> }
>>
>> // Note the dependency between the vertices in the graph.
>> graph.markDependency(splitHead, splitTail);
>> }
>>
>>
>> my question is , since $vertex is e's tail, why graph.getEdge(vertex,
>> splitTail).addNodePath(headPath)?
>>
>> it seems like crunch reverse the edeg e?
>>
>>
>> --
>>
>> Jing Chen HPCC.ICT.AC China
>>
>
>
--
陈竞，中科院计算技术研究所，高性能计算机中心
Jing Chen HPCC.ICT.AC China
|