spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravi Agrawal <Ravi.Agra...@talentica.com>
Subject Unable to kill spark app gracefully. Unable to stop driver in cluster mode
Date Fri, 17 Jun 2016 13:09:56 GMT
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;}
--></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">Hi,</span><br>
</p>
<div style="color: rgb(0, 0, 0);">
<div>
<div id="divtagdefaultwrapper" style="font-size:12pt; color:#000000; background-color:#FFFFFF;
font-family:Calibri,Arial,Helvetica,sans-serif">
<p><br>
</p>
<p>While working on Spark 1.6.1, I ran into an issue with closing the Spark app.</p>
<p><br>
</p>
<p>I tried it with deploy-mode as&nbsp;client as well as cluster:</p>
<p><br>
</p>
<p><br>
</p>
<p><b><span style="font-size:14pt">Firstly,&nbsp;</span><span
style="font-size:14pt">deploy-mode : client</span></b></p>
<p><br>
</p>
<p><u>Ran the app using below command:</u></p>
<p><span>/usr/local/spark/spark-1.6.1-bin-hadoop2.6/bin/spark-submit --supervise
--class &lt;Qualified class name&gt; &lt;Jar path&gt;</span><br>
</p>
<p><span><br>
</span></p>
<p><span><u>Killed from UI by clicking the 'kill' link:</u></span></p>
<p><span>App got removed from UI but it was still running on the server.</span></p>
<p><span><br>
</span></p>
<p><span><u>Killed using below command:</u></span></p>
<p><span><span>bin/spark-class org.apache.spark.deploy.Client kill spark://raviagrawal:7077
app-20160617181402-0005</span><br>
</span></p>
<p><span><span>The above command did not work. Gave below message on console:</span></span></p>
<p><span><span><span>ClientEndpoint: Driver app-20160617181402-0005
has already finished or does not exist</span><br>
</span></span></p>
<p><span><span><span>The app was still running</span></span></span></p>
<p><span><span><span><br>
</span></span></span></p>
<p><span><span><span><br>
</span></span></span></p>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe
UI Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px"><b><span
style="font-size:14pt">Secondly,&nbsp;</span></b></span><span
style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI Emoji',NotoColorEmoji,'Segoe
UI Symbol','Android Emoji',EmojiSymbols; font-size:16px"><b><span style="font-size:14pt">deploy-mode
 : cluster</span></b></span><br>
</p>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe
UI Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px"><br>
</span></p>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe
UI Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px"></span></p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
<u>Ran the app using below command:</u></p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
<span>/usr/local/spark/spark-1.6.1-bin-hadoop2.6/bin/spark-submit &nbsp;--master
spark://raviagrawal:7077 --deploy-mode cluster --supervise --class &lt;Qualified class
name&gt; &lt;Jar path&gt;</span><br>
</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
<br>
</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
<u>Killed from UI by clicking the 'kill' link:</u></p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
App got removed from UI but it was still running on the server.</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
Unable to kill driver from UI</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
<br>
</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
<u>Killed using below command:</u></p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
<span>bin/spark-class org.apache.spark.deploy.Client kill spark://raviagrawal:7077 driver-20160617181915-0003</span></p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
The above command did not work. Gave below message on console:</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
</p>
<div>16/06/17 18:20:04 INFO ClientEndpoint: Kill request for driver-20160617181915-0003
submitted</div>
<div>16/06/17 18:20:04 INFO ClientEndpoint: ... waiting before polling master for driver
state</div>
<div>16/06/17 18:20:09 INFO ClientEndpoint: ... polling master for driver state</div>
<div>16/06/17 18:20:09 INFO ClientEndpoint: State of driver-20160617181915-0003 is RUNNING</div>
<div>16/06/17 18:20:09 INFO ClientEndpoint: Driver running on 172.19.19.105:55454 (worker-20160617134955-172.19.19.105-55454)</div>
<p></p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
<br>
</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
<br>
</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
I went through the resources available online but could not find a solution to the problem.</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
I can kill the process using kill -9 pid, but do not want to use it. I want to kill the app
gracefully.</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
Also found <a href="https://issues.apache.org/jira/browse/SPARK-11588">this JIRA</a>
mentioning almost the same issue, but the status of that has been changed to Invalid as several
questions were asked in it.</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
<br>
</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
Can you please help me on this?</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
<br>
</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
Thanks,</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
-Ravi.</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
Talentica Software India Pvt. Ltd.</p>
<p style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe UI
Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px">
<br>
</p>
<br>
<p></p>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe
UI Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px"><br>
</span></p>
<p><span style="font-family:Calibri,Arial,Helvetica,sans-serif,'Apple Color Emoji','Segoe
UI Emoji',NotoColorEmoji,'Segoe UI Symbol','Android Emoji',EmojiSymbols; font-size:16px"><br>
</span></p>
<p></p>
</div>
</div>
</div>
</div>
</body>
</html>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message