spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vipul Pandey <vipan...@gmail.com>
Subject Spark (trunk/yarn) on CDH4.3.0.2 - YARN
Date Fri, 06 Sep 2013 22:37:25 GMT
I'm unable to successfully run the SparkPi example in my YARN cluster.

I did whatever has been specified here (didn't change anything
anywhere) : http://spark.incubator.apache.org/docs/0.7.0/running-on-yarn.html

and added HADOOP_CONF_DIR as well. (btw, on sbt/sbt assembly - the jar
file it generates is spark-core-assembly-*0.6.0*.jar)


I get the following exception in my container :


Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
	at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135)
	at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:103)
	at spark.deploy.yarn.ApplicationMaster.registerApplicationMaster(ApplicationMaster.scala:123)
	at spark.deploy.yarn.ApplicationMaster.spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:52)
	at spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:42)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
	at spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:40)
	at spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:340)
	at spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: com.google.protobuf.ServiceException: java.io.IOException:
Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Message missing
required fields: callId, status; Host Details : local host is:
"rd17d01ls-vm0109.rd.geo.apple.com/17.134.172.65"; destination host
is: "rd17d01ls-vm0110.rd.geo.apple.com":8030;
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:212)
	at $Proxy7.registerApplicationMaster(Unknown Source)
	at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:100)
	... 9 more
Caused by: java.io.IOException: Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Message missing
required fields: callId, status; Host Details : local host is:
"rd17d01ls-vm0109.rd.geo.apple.com/17.134.172.65"; destination host
is: "rd17d01ls-vm0110.rd.geo.apple.com":8030;
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:761)
	at org.apache.hadoop.ipc.Client.call(Client.java:1239)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
	... 11 more
Caused by: com.google.protobuf.InvalidProtocolBufferException: Message
missing required fields: callId, status
	at com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:81)
	at org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto$Builder.buildParsed(RpcPayloadHeaderProtos.java:1094)
	at org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto$Builder.access$1300(RpcPayloadHeaderProtos.java:1028)
	at org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcPayloadHeaderProtos.java:986)
	at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:946)
	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:844)





Any solutions anyone?

Mime
View raw message