spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: org.apache.spark.SparkException Error sending message
Date Mon, 16 Mar 2015 07:02:49 GMT
Not sure if this will help, but can you try setting the following:

set("spark.core.connection.ack.wait.timeout","6000")


Thanks
Best Regards

On Sat, Mar 14, 2015 at 4:08 AM, Chen Song <chen.song.82@gmail.com> wrote:

> When I ran Spark SQL query (a simple group by query) via hive support, I
> have seen lots of failures in map phase.
>
> I am not sure if that is specific to Spark SQL or general.
>
> Any one has seen such errors before?
>
> java.io.IOException: org.apache.spark.SparkException: Error sending
> message [message = GetLocations(broadcast_9_piece0)]
>
> java.io.IOException: org.apache.spark.SparkException: Error sending message [message
= GetLocations(broadcast_9_piece0)]
> 	at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1093)
> 	at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:164)
> 	at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
> 	at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
> 	at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:87)
> 	at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
> 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:61)
> 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:56)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.spark.SparkException: Error sending message [message = GetLocations(broadcast_9_piece0)]
> 	at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:201)
> 	at org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:221)
> 	at org.apache.spark.storage.BlockManagerMaster.getLocations(BlockManagerMaster.scala:70)
> 	at org.apache.spark.storage.BlockManager.doGetRemote(BlockManager.scala:592)
> 	at org.apache.spark.storage.BlockManager.getRemoteBytes(BlockManager.scala:587)
> 	at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.org$apache$spark$broadcast$TorrentBroadcast$$anonfun$$getRemote$1(TorrentBroadcast.scala:126)
> 	at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$1.apply(TorrentBroadcast.scala:136)
> 	at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$1.apply(TorrentBroadcast.scala:136)
> 	at scala.Option.orElse(Option.scala:257)
> 	at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply$mcVI$sp(TorrentBroadcast.scala:136)
> 	at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:119)
> 	at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:119)
> 	at scala.collection.immutable.List.foreach(List.scala:318)
> 	at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$readBlocks(TorrentBroadcast.scala:119)
> 	at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:174)
> 	at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1090)
> 	... 12 more
> Caused by: java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]
> 	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> 	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> 	at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> 	at scala.concurrent.Await$.result(package.scala:107)
> 	at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:187)
> 	... 27 more
>
>
> --
> Chen Song
>
>

Mime
View raw message