spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: How to run spark shell using YARN
Date Tue, 13 Mar 2018 01:13:33 GMT
Hi,

I read that doc several times now. I am stuck with the below error message
when I run ./spark-shell --master yarn --deploy-mode client.

I have my HADOOP_CONF_DIR set to /usr/local/hadoop-2.7.3/etc/hadoop and
SPARK_HOME set to /usr/local/spark on all 3 machines (1 node for Resource
Manager and NameNode, 2 Nodes for Node Manager and DataNodes).

Any idea?



*18/03/13 00:19:13 INFO LineBufferedStream: stdout:
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
/user/centos/.sparkStaging/application_1520898664848_0003/__spark_libs__2434167523839846774.zip
could only be replicated to 0 nodes instead of minReplication (=1).  There
are 2 datanode(s) running and no node(s) are excluded in this operation.*


18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
java.security.AccessController.doPrivileged(Native Method)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
javax.security.auth.Subject.doAs(Subject.java:422)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
18/03/13


Thanks!


On Mon, Mar 12, 2018 at 4:46 PM, Marcelo Vanzin <vanzin@cloudera.com> wrote:

> That's not an error, just a warning. The docs [1] have more info about
> the config options mentioned in that message.
>
> [1] http://spark.apache.org/docs/latest/running-on-yarn.html
>
> On Mon, Mar 12, 2018 at 4:42 PM, kant kodali <kanth909@gmail.com> wrote:
> > Hi All,
> >
> > I am trying to use YARN for the very first time. I believe I configured
> all
> > the resource manager and name node fine. And then I run the below command
> >
> > ./spark-shell --master yarn --deploy-mode client
> >
> > I get the below output and it hangs there forever (I had been waiting
> over
> > 10 minutes)
> >
> > 18/03/12 23:36:32 WARN Client: Neither spark.yarn.jars nor
> > spark.yarn.archive is set, falling back to uploading libraries under
> > SPARK_HOME.
> >
> > Any idea?
> >
> > Thanks!
>
>
>
> --
> Marcelo
>

Mime
View raw message