Some corrections.

On Fri, Sep 26, 2014 at 5:32 PM, praveen seluka <praveen.seluka@gmail.com> wrote:
Can someone explain the motivation behind passing executorAdded event to DAGScheduler ? DAGScheduler does submitWaitingStages when executorAdded method is called by TaskSchedulerImpl. I see some issue in the below code,

TaskSchedulerImpl.scala code
if (!executorsByHost.contains(o.host)) {
        executorsByHost(o.host) = new HashSet[String]()
        executorAdded(o.executorId, o.host)
        newExecAvail = true
      }

Note that executorAdded is called only when there is a new host and not for every new executor. For instance, there can be two executors in the same host and in this case the DAGscheduler is notified only once. If this is indeed an issue, I would like to submit a patch for this quickly. [cc Andrew Or]

- Praveen