spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Praveen Seluka <>
Subject Getting started : Spark on YARN issue
Date Thu, 19 Jun 2014 13:04:03 GMT
I am trying to run Spark on YARN. I have a hadoop 2.2 cluster (YARN  +
HDFS) in EC2. Then, I compiled Spark using Maven with 2.2 hadoop profiles.
Now am trying to run the example Spark job . (In Yarn-cluster mode).

>From my *local machine. *I have setup HADOOP_CONF_DIR environment variable

➜  spark git:(master) ✗ /bin/bash -c "./bin/spark-submit --class
org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors 2
--driver-memory 2g --executor-memory 2g --executor-cores 1
examples/target/scala-2.10/spark-examples_*.jar 10"
14/06/19 14:59:39 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/06/19 14:59:39 INFO client.RMProxy: Connecting to ResourceManager at
14/06/19 14:59:41 INFO yarn.Client: Got Cluster metric info from
ApplicationsManager (ASM), number of NodeManagers: 1
14/06/19 14:59:41 INFO yarn.Client: Queue info ... queueName: default,
queueCurrentCapacity: 0.0, queueMaxCapacity: 1.0,
      queueApplicationCount = 0, queueChildQueueCount = 0
14/06/19 14:59:41 INFO yarn.Client: Max mem capabililty of a single
resource in this cluster 12288
14/06/19 14:59:41 INFO yarn.Client: Preparing Local resources
14/06/19 14:59:42 WARN hdfs.BlockReaderLocal: The short-circuit local reads
feature cannot be used because libhadoop cannot be loaded.
14/06/19 14:59:43 INFO yarn.Client: Uploading
to hdfs://
14/06/19 15:00:45 INFO hdfs.DFSClient: Exception in createBlockOutputStream 60000 millis timeout while
waiting for channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending remote=/]
14/06/19 15:00:45 INFO hdfs.DFSClient: Abandoning
14/06/19 15:00:46 INFO hdfs.DFSClient: Excluding datanode
14/06/19 15:00:46 WARN hdfs.DFSClient: DataStreamer Exception

Its able to talk to Resource Manager
Then it puts the example.jar file to HDFS and it fails. Its trying to write
to datanode. I verified that 50010 port is accessible through local
machine. Any idea whats the issue here ?
One thing thats suspicious is */
<> - it looks like its trying to connect using
private IP. If so, how can I resolve this to use public IP.*


View raw message