spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akshay Bhardwaj <akshay.bhardwaj1...@gmail.com>
Subject Re: Why my spark job STATE--> Running FINALSTATE --> Undefined.
Date Wed, 12 Jun 2019 05:49:43 GMT
Hi Shyam,

It will be good if you mention what are you using the --master url as? Is
it running on YARN, Mesos or Spark cluster?

However, I faced such an issue in my earlier trials with spark, in which I
created connections with a lot of external databases like Cassandra within
the Driver (or main program of my app).
After the job completed, my Main program/driver task never finished, after
debugging I found out to be the reason as open sessions with Cassandra.
Closing out those connections at the end of my main program helped resolve
the problem. As you can guess, this issue was then irrespective of the
Cluster manager used.


Akshay Bhardwaj
+91-97111-33849


On Tue, Jun 11, 2019 at 7:41 PM Shyam P <shyamabigdata@gmail.com> wrote:

> Hi,
> Any clue why spark job goes into UNDEFINED state ?
>
> More detail are in the url.
>
> https://stackoverflow.com/questions/56545644/why-my-spark-sql-job-stays-in-state-runningfinalstatus-undefined
>
>
> Appreciate your help.
>
> Regards,
> Shyam
>

Mime
View raw message