spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis.gospodne...@gmail.com>
Subject Spark @ EC2: Futures timed out & Ask timed out
Date Mon, 16 Mar 2015 21:56:09 GMT
Hi,

I've been trying to run a simple SparkWordCount app on EC2, but it looks
like my apps are not succeeding/completing.  I'm suspecting some sort of
communication issue.  I used the SparkWordCount app from
http://blog.cloudera.com/blog/2014/04/how-to-run-a-simple-apache-spark-app-in-cdh-5/


Digging through logs I found this:

 15/03/16 21:28:20 INFO Utils: Successfully started service
'driverPropsFetcher' on port 58123.


 Exception in thread "main" java.lang.reflect.UndeclaredThrowableException



         at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1563)


         at
org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:60)


         at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:115)


         at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:163)


         at
org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)


* Caused by: java.util.concurrent.TimeoutException: Futures timed out after
[30 seconds] *


         at
scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)


         at
scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)


         at
scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)


         at
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)


         at scala.concurrent.Await$.result(package.scala:107)



         at
org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:127)


         at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)


         at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)


         at java.security.AccessController.doPrivileged(Native Method)



         at javax.security.auth.Subject.doAs(Subject.java:415)



         at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)


         ... 4 more


Or exceptions like:

*Caused by: akka.pattern.AskTimeoutException: Ask timed out on
[ActorSelection[Anchor(akka.tcp://sparkDriver@ip-10-111-222-111.ec2.internal:58360/),
Path(/user/CoarseGrainedScheduler)]] after [30000 ms]  *

         at
akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333)


         at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)



         at
scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)


         at
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)


         at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)


         at
akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)


         at
akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)


         at
akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)


         at java.lang.Thread.run(Thread.java:745)


This is in EC2 and I have ports 22, 7077, 8080, and 8081 open to any source.
But maybe I need to do something, too?

I do see Master sees Workers and Workers do connect to the Master.

I did run this in spark-shell, and it runs without problems;
scala> val something = sc.parallelize(1 to 10000000).collect().filter(_<1000

This is how I submitted the job (on the Master machine):

$ spark-1.2.1-bin-hadoop2.4/bin/spark-submit --class
com.cloudera.sparkwordcount.SparkWordCount --executor-memory 256m --master
spark://ip-10-171-32-62:7077
wc-spark/target/sparkwordcount-0.0.1-SNAPSHOT.jar /usr/share/dict/words 0

Any help would be greatly appreciated.

Thanks,
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/

Mime
View raw message