spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacob Chacko - Catalyst Consulting <>
Subject akka.remote.transport.netty.NettyTransport
Date Tue, 07 Oct 2014 15:09:00 GMT
Hi All

I have one master and one worker on AWS (amazon web service) and am running spark map reduce
code provided on the link
We are using Spark version 1.0.2

Word Count
val file = spark.textFile("hdfs://...")
val counts = file.flatMap(line => line.split(" "))
                 .map(word => (word, 1))
                 .reduceByKey(_ + _)

To get this working I have compiled a jar file.
spark-submit --master spark://ip-172-31-24-183.ec2.internal:7077 --class Sample /home/ec2-user/scalatest/target/scala-2.11/test_big_2.11-0.1.jar

when I  run the spark job it gives me error: Connection refused -

  1.  There is not result shown on the master
  2.  Error on worker is as below

14/10/07 10:16:16 ERROR EndpointWriter: AssociationError [akka.tcp://sparkWorker@ip-172-31-27-51.ec2.internal:39142]
-> [akka.tcp://sparkExecutor@ip-172-31-27-51.ec2.internal:38752]: Error [Association failed
with [akka.tcp://sparkExecutor@ip-172-31-27-51.ec2.internal:38752]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkExecutor@ip-172-31-27-51.ec2.internal:38752]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection
refused: ip-172-31-27-51.ec2.internal/

 Could some one help clarify if this is a

  *   spark issue
  *   issue with the jar file
  *   or if slave cannot connect to master or itself or some port ?

Please help


View raw message