spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <van...@cloudera.com>
Subject Re: How to run spark shell using YARN
Date Tue, 13 Mar 2018 01:21:47 GMT
Looks like you either have a misconfigured HDFS service, or you're
using the wrong configuration on the client.

BTW, as I said in the previous response, the message you saw initially
is *not* an error. If you're just trying things out, you don't need to
do anything and Spark should still work.

On Mon, Mar 12, 2018 at 6:13 PM, kant kodali <kanth909@gmail.com> wrote:
> Hi,
>
> I read that doc several times now. I am stuck with the below error message
> when I run ./spark-shell --master yarn --deploy-mode client.
>
> I have my HADOOP_CONF_DIR set to /usr/local/hadoop-2.7.3/etc/hadoop and
> SPARK_HOME set to /usr/local/spark on all 3 machines (1 node for Resource
> Manager and NameNode, 2 Nodes for Node Manager and DataNodes).
>
> Any idea?
>
>
>
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /user/centos/.sparkStaging/application_1520898664848_0003/__spark_libs__2434167523839846774.zip
> could only be replicated to 0 nodes instead of minReplication (=1).  There
> are 2 datanode(s) running and no node(s) are excluded in this operation.
>
>
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> java.security.AccessController.doPrivileged(Native Method)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> javax.security.auth.Subject.doAs(Subject.java:422)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> 18/03/13
>
>
> Thanks!
>
>
> On Mon, Mar 12, 2018 at 4:46 PM, Marcelo Vanzin <vanzin@cloudera.com> wrote:
>>
>> That's not an error, just a warning. The docs [1] have more info about
>> the config options mentioned in that message.
>>
>> [1] http://spark.apache.org/docs/latest/running-on-yarn.html
>>
>> On Mon, Mar 12, 2018 at 4:42 PM, kant kodali <kanth909@gmail.com> wrote:
>> > Hi All,
>> >
>> > I am trying to use YARN for the very first time. I believe I configured
>> > all
>> > the resource manager and name node fine. And then I run the below
>> > command
>> >
>> > ./spark-shell --master yarn --deploy-mode client
>> >
>> > I get the below output and it hangs there forever (I had been waiting
>> > over
>> > 10 minutes)
>> >
>> > 18/03/12 23:36:32 WARN Client: Neither spark.yarn.jars nor
>> > spark.yarn.archive is set, falling back to uploading libraries under
>> > SPARK_HOME.
>> >
>> > Any idea?
>> >
>> > Thanks!
>>
>>
>>
>> --
>> Marcelo
>
>



-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message