spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacob Chacko - Catalyst Consulting <jacob.cha...@catalystconsulting.be>
Subject akka.remote.transport.netty.NettyTransport
Date Tue, 07 Oct 2014 15:09:00 GMT
Hi All

I have one master and one worker on AWS (amazon web service) and am running spark map reduce
code provided on the link https://spark.apache.org/examples.html
We are using Spark version 1.0.2

Word Count
val file = spark.textFile("hdfs://...")
val counts = file.flatMap(line => line.split(" "))
                 .map(word => (word, 1))
                 .reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://...")

To get this working I have compiled a jar file.
spark-submit --master spark://ip-172-31-24-183.ec2.internal:7077 --class Sample /home/ec2-user/scalatest/target/scala-2.11/test_big_2.11-0.1.jar


when I  run the spark job it gives me error: Connection refused -


  1.  There is not result shown on the master
  2.  Error on worker is as below

14/10/07 10:16:16 ERROR EndpointWriter: AssociationError [akka.tcp://sparkWorker@ip-172-31-27-51.ec2.internal:39142]
-> [akka.tcp://sparkExecutor@ip-172-31-27-51.ec2.internal:38752]: Error [Association failed
with [akka.tcp://sparkExecutor@ip-172-31-27-51.ec2.internal:38752]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkExecutor@ip-172-31-27-51.ec2.internal:38752]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection
refused: ip-172-31-27-51.ec2.internal/172.31.27.51:38752
]

 Could some one help clarify if this is a

  *   spark issue
  *   issue with the jar file
  *   or if slave cannot connect to master or itself or some port ?

Please help

Thanks
Jacob




Mime
View raw message