spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bsikander <>
Subject Re: Programmatically get status of job (WAITING/RUNNING)
Date Thu, 07 Dec 2017 19:40:47 GMT
Marcelo Vanzin wrote
> I'm not sure I follow you here. This is something that you are
> defining, not Spark.

Yes, you are right. In my code, 
1) my notion of RUNNING is that both driver + executors are in RUNNING
2) my notion of WAITING is if any one of driver/executor is in WAITING

- SparkLauncher provides me the details about the "driver".
- SparkListener provides me the details about the "executor" using

I want to combine both SparkLauncher + SparkListener to achieve my view of

The only thing confusing me here is that I don't know how Spark internally
converts applications from WAITING to RUNNING state.
For example, if an application wanted 4 executors
(spark.executor.instances=4) but the spark cluster can only provide 1
executor. This means that I will only receive 1 onExecutorAdded event. Will
the application state change to RUNNING (even if 1 executor was allocated)?

If I am clear on this logic I can implement my feature.

Sent from:

To unsubscribe e-mail:

View raw message