tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Nicoara <adn...@microsoft.com.INVALID>
Subject RE: TEZ-1190 status
Date Thu, 23 Aug 2018 00:05:53 GMT
As a small follow up, here are parts of the code that I stumbled upon.
Each edge that has a vertex with 0 tasks as its destination, says that no routing of events
is required:
https://github.com/apache/tez/blob/4b9a7be1b98cff00c44e7d3ffb2486bb59ca6804/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/Edge.java#L263-L264

That causes the EdgeManager code to be skipped:
https://github.com/apache/tez/blob/4b9a7be1b98cff00c44e7d3ffb2486bb59ca6804/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/Edge.java#L487-L527

Is there another mechanism that the VertexManager uses to take action on data movement events,
other than routing through the edge manager?

> -----Original Message-----
> From: Adrian Nicoara <adnico@microsoft.com.INVALID>
> Sent: Monday, August 13, 2018 1:43 PM
> To: dev@tez.apache.org; Hitesh Sharma <hitesh@microsoft.com>; . Anupam
> <anupam@microsoft.com>
> Subject: RE: TEZ-1190 status
> 
> > -----Original Message-----
> > From: Gopal Vijayaraghavan <gopalv@apache.org>
> > Sent: Friday, August 10, 2018 2:20 PM
> > To: dev@tez.apache.org; Hitesh Sharma <hitesh@microsoft.com>; .
> Anupam
> > <anupam@microsoft.com>
> > Subject: Re: TEZ-1190 status
> >
> > As far as I know, that patch + design has been abandoned.
> >
> > There was a discussion about adding it without fundamentally changing
> > the DAGImpl (and task recovery etc).
> >
> > The VertexManager already allows you to have a vertex without a task
> > in it, without giving up on being a participant in the data movement events.
> 
> This sounds very promising. Can you point me to any tests, or sample code,
> or the implementation code I should be looking at?
> 
> > However, there are no immediate takers for the new approach (which is
> > additive, so needs no significant refactoring changes) and this
> > feature did not have anyone waiting for it.
> 
> If it can be done with the above suggestion, I can pick up this work, if it is OK.
Mime
View raw message