spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex <siri8...@gmail.com>
Subject Re: keep or remove sc.stop() coz of RpcEnv already stopped error
Date Mon, 13 Mar 2017 14:53:22 GMT
Hi ,
I am using spark-1.6 how to ignore this warning because of this Illegal
state exception my production jobs which are scheduld are showing completed
abnormally... I cant even handle exception as after sc.stop if i try to
execute any code again this exception will come from catch block.. so i
removed sc.stop completely and let it exit on its own is it okay?

On Mon, Mar 13, 2017 at 7:01 PM, Yong Zhang <java8964@hotmail.com> wrote:

> What version of Spark you are using?
>
>
> Based on Spark-12967, it is fixed on Spark 2.0 and later. If you are using
> Spark 1.x, you can ignore this Warning. It shouldn't affect any functions.
>
>
> Yong
>
> ------------------------------
> *From:* nancy henry <nancyhenry6542@gmail.com>
> *Sent:* Monday, March 13, 2017 7:08 AM
> *To:* user @spark
> *Subject:* keep or remove sc.stop() coz of RpcEnv already stopped error
>
>
>  Hi Team,
>
>
> getting this error if we put sc.stop() in application..
>
>
>
> can we remove it from application but i read if yu dont explicitly stop
> using sc.stop the yarn application will not get registered in history
> service.. SO what to do ?
>
>
>  WARN Dispatcher: Message RemoteProcessDisconnected dropped.
>
> java.lang.IllegalStateException: RpcEnv already stopped.
>
>         at org.apache.spark.rpc.netty.Dispatcher.postMessage(
> Dispatcher.scala:159)
>
>         at org.apache.spark.rpc.netty.Dispatcher.postToAll(
> Dispatcher.scala:109)
>
>         at org.apache.spark.rpc.netty.NettyRpcHandler.
> connectionTerminated(NettyRpcEnv.scala:630)
>
>         at org.apache.spark.network.server.TransportRequestHandler.
> channelUnregistered(TransportRequestHandler.java:94)
>
>
>
>
>

Mime
View raw message