spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: akka.remote.transport.netty.NettyTransport
Date Thu, 09 Oct 2014 07:50:22 GMT
This issue is related to your cluster. Can you paste your spark-env.sh?

Also you can try starting a spark-shell like $SPARK_HOME/bin/spark-shell
--master=*spark://ip-172-31-24-183.ec2.internal:7077* and try the same code
in it.

Is this the same URI spark://ip-172-31-24-183.ec2.internal:7077 shown in
your webUI http://master-host:8080(top left)?

Thanks
Best Regards

On Tue, Oct 7, 2014 at 8:39 PM, Jacob Chacko - Catalyst Consulting <
jacob.chacko@catalystconsulting.be> wrote:

>  Hi All
>
>  I have one master and one worker on AWS (amazon web service) and am
> running spark map reduce code provided on the link
> https://spark.apache.org/examples.html
> We are using Spark version 1.0.2
>
>  Word Count
> val file = spark.textFile("hdfs://...")
> val counts = file.flatMap(line => line.split(" "))
>                  .map(word => (word, 1))
>                  .reduceByKey(_ + _)
> counts.saveAsTextFile("hdfs://...")
>
> To get this working I have compiled a jar file.
> spark-submit --master spark://ip-172-31-24-183.ec2.internal:7077 --class
> Sample /home/ec2-user/scalatest/target/scala-2.11/test_big_2.11-0.1.jar
>
>
>  when I  run the spark job it gives me error: Connection refused -
>
>
>    1. There is not result shown on the master
>    2. Error on worker is as below
>
>
>  14/10/07 10:16:16 ERROR EndpointWriter: AssociationError
> [akka.tcp://sparkWorker@ip-172-31-27-51.ec2.internal:39142] ->
> [akka.tcp://sparkExecutor@ip-172-31-27-51.ec2.internal:38752]: Error
> [Association failed with
> [akka.tcp://sparkExecutor@ip-172-31-27-51.ec2.internal:38752]] [
> akka.remote.EndpointAssociationException: Association failed with
> [akka.tcp://sparkExecutor@ip-172-31-27-51.ec2.internal:38752]
> Caused by:
> akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
> Connection refused: ip-172-31-27-51.ec2.internal/172.31.27.51:38752
> ]
>
>   Could some one help clarify if this is a
>
>    - spark issue
>    - issue with the jar file
>    - or if slave cannot connect to master or itself or some port ?
>
>
>  Please help
>
>  Thanks
> Jacob
>
>
>
>

Mime
View raw message