spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amjad ALSHABANI <ashshab...@gmail.com>
Subject Re: spark-shell with Yarn failed
Date Sat, 18 Jul 2015 18:46:02 GMT
Does anybody have any idea about the error I m having.. I am really
clueless... And appreciate any idea :)

Thanks in advance

Amjad
On Jul 17, 2015 5:37 PM, "Amjad ALSHABANI" <ashshabani@gmail.com> wrote:

> Hello,
>
> First of all I m a newbie in Spark ,
>
> I m trying to start the spark-shell with yarn cluster by running:
>
> $ spark-shell --master yarn-client
>
> Sometimes it goes well, but most of the time I got an error:
>
> Container exited with a non-zero exit code 10
> Failing this attempt. Failing the application.
>          ApplicationMaster host: N/A
>          ApplicationMaster RPC port: -1
>          queue: default
>          start time: 1437145851944
>          final status: FAILED
>          tracking URL:
> http://My-HadoopServer:50080/cluster/app/application_1437033338028_0030
>          user: hadoop
> org.apache.spark.SparkException: Yarn application has already ended! It
> might have been killed or unable to launch application master.
>         at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:115)
> ....
> ....
> ....
>
> searching in the yarn logs I got this log
>
> $ yarn logs -applicationId application_1437033338028_0030
> 2015-07-17 17:11:03,961 - INFO
> [sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3@74]
> - Starting remoting
> 2015-07-17 17:11:04,200 - ERROR
> [sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$1@65]
> - failed to bind to My-HadoopServer/10.98.105.11:0, shutting down Netty
> transport
> 2015-07-17 17:11:04,210 - WARN  [main:Logging$class@71] - Service
> 'sparkYarnAM' could not bind on port 0. Attempting port 1.
> ...
> ...
> ...
> 2015-07-17 17:11:05,123 - ERROR [main:Logging$class@96] - Uncaught
> exception:
> java.net.BindException: Failed to bind to:
> My-HadoopServer/HadoopServerIP:0: Service 'sparkYarnAM' failed after 16
> retries!
>         at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
>         at
> akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:393)
>         at
> akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389)
>         at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
> ...
> ....
> ....
>
>
>
> I m using Spark 1.3, Hadoop 2.6 ,
>
>  and in spark-env.sh it points to my hadoop configuration:
>
> export HADOOP_CONF_DIR=/usr/hdp/2.2.4.4-16/hadoop/conf
>
>
> Is this probleme coming from spark configuration or yarn configuration (or
> spark with yarn confs)????
>
> Any Ideas??
>
>
>
> Amjad
>
>

Mime
View raw message