spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Zhang <zjf...@gmail.com>
Subject Re: Event logging not working when worker machine terminated
Date Wed, 09 Sep 2015 03:18:07 GMT
What cluster mode do you use ? Standalone/Yarn/Mesos ?


On Wed, Sep 9, 2015 at 11:15 AM, David Rosenstrauch <darose@darose.net>
wrote:

> Our Spark cluster is configured to write application history event logging
> to a directory on HDFS.  This all works fine.  (I've tested it with Spark
> shell.)
>
> However, on a large, long-running job that we ran tonight, one of our
> machines at the cloud provider had issues and had to be terminated and
> replaced in the middle of the job.
>
> The job completed correctly, and shows in state FINISHED in the "Completed
> Applications" section of the Spark GUI.  However, when I try to look at the
> application's history, the GUI says "Application history not found" and
> "Application ... is still in progress".
>
> The reason appears to be the machine that was terminated.  When I click on
> the executor list for that job, Spark is showing the executor from the
> terminated machine as still in state RUNNING.
>
> Any solution/workaround for this?  BTW, I'm running Spark v1.3.0.
>
> Thanks,
>
> DR
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>


-- 
Best Regards

Jeff Zhang

Mime
View raw message