spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nan Zhu <zhunanmcg...@gmail.com>
Subject Re: executorAdded event to DAGScheduler
Date Fri, 26 Sep 2014 12:35:42 GMT
just a quick reply, we cannot start two executors in the same host for a single application
in the standard deployment (one worker per machine)  

I’m not sure if it will create an issue when you have multiple workers in the same host,
as submitWaitingStages is called everywhere and I never try such a deployment mode

Best,  

--  
Nan Zhu


On Friday, September 26, 2014 at 8:02 AM, praveen seluka wrote:

> Can someone explain the motivation behind passing executorAdded event to DAGScheduler
? DAGScheduler does submitWaitingStages when executorAdded method is called by TaskSchedulerImpl.
I see some issue in the below code,
>  
> TaskSchedulerImpl.scala code
> if (!executorsByHost.contains(o.host)) {
>         executorsByHost(o.host) = new HashSet[String]()
>         executorAdded(o.executorId, o.host)
>         newExecAvail = true
>       }
>  
>  
> Note that executorAdded is called only when there is a new host and not for every new
executor. For instance, there can be two executors in the same host and in this case. (But
DAGScheduler executorAdded is notified only for new host - so only once in this case). If
this is indeed an issue, I would like to submit a patch for this quickly. [cc Andrew Or]
>  
> - Praveen
>  
>  


Mime
View raw message