spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chandu <chandra.vedant...@gmail.com>
Subject Spark Standalone Mode, application runs, but executor is killed
Date Fri, 26 Jan 2018 03:09:24 GMT
Hi,
I tried my question @ stackoverlfow.com (
https://stackoverflow.com/questions/48445145/spark-standalone-mode-application-runs-but-executor-is-killed-with-exitstatus),
yet to be answere, so thought I will tru the user group.

I am new to Apache Spark and was trying to run the example Pi Calculation
application on my local spark setup (using Standalone Cluster). Both the
Master, Slave and Driver are running on my local machine.

What I am noticing is that, the PI is calculated successfully, however in
the slave logs I see that the Worker/Executor is being killed with
exitStatus 1. I do not see any errors/exceptions logged to the console
otherwise. I tried finding help on similar issue, but most of the search
hits were referring to exitStatus 137 etc. (e.g: Spark application kills
executor
https://stackoverflow.com/questions/40910952/spark-application-kills-executor
<https://stackoverflow.com/questions/40910952/spark-application-kills-executor>  

I have failed miserably to understand why the Worker is being killed
instead of completing the execution with 'EXITED' state. I think it's
related to how I am executing the app, but am not quite clear what am I
doing wrong.

The code and logs are available @
https://gist.github.com/Chandu/a83c13c045f1d1b480d8839e145b2749
<https://gist.github.com/Chandu/a83c13c045f1d1b480d8839e145b2749>   (trying
to
keep the email content short)

I wantd to understand that if my assumption of an Executor should have a
state of Exited when there are no errors in execution or is it always set
as KILLED when a spark job is completed?

I tried to understand the flow looking at the source code and with my
limited understanding of the code, I found that the Executor would always
end up with KILLED status (most likely my conclusion is wrong) based on the
code @
https://github.com/apache/spark/blob/39e2bad6a866d27c3ca594d15e574a1da3ee84cc/core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala#L118
<https://github.com/apache/spark/blob/39e2bad6a866d27c3ca594d15e574a1da3ee84cc/core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala#L118>
 

Can someone guide me on identifying the root cause for this issue or if my
assumption of the Exectuor having a status of EXITED at the end of
execution is not correct?






--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message