spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From praveen seluka <praveen.sel...@gmail.com>
Subject Re: executorAdded event to DAGScheduler
Date Fri, 26 Sep 2014 12:03:17 GMT
Some corrections.

On Fri, Sep 26, 2014 at 5:32 PM, praveen seluka <praveen.seluka@gmail.com>
wrote:

> Can someone explain the motivation behind passing executorAdded event to
> DAGScheduler ? *DAGScheduler *does *submitWaitingStages *when *executorAdded
> *method is called by *TaskSchedulerImpl*. I see some issue in the below
> code,
>
> *TaskSchedulerImpl.scala code*
> if (!executorsByHost.contains(o.host)) {
>         executorsByHost(o.host) = new HashSet[String]()
>         executorAdded(o.executorId, o.host)
>         newExecAvail = true
>       }
>
> Note that executorAdded is called only when there is a new host and not
> for every new executor. For instance, there can be two executors in the
> same host and in this case the DAGscheduler is notified only once. If this
> is indeed an issue, I would like to submit a patch for this quickly. [cc
> Andrew Or]
>
> - Praveen
>
>
>

Mime
View raw message