spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From suman bharadwaj <suman....@gmail.com>
Subject SPARK protocol buffer issue. Need Help
Date Mon, 20 Jan 2014 21:05:11 GMT
*Hi,*

*I'm new to spark. And I was trying to read a file residing in HDFS. And
perform some basic actions on this dataset. See below the code i used:*

*object Hbase {*
*  def main(args: Array[String]) {*
*    val sc = new
SparkContext("spark://<servername>:<portno>","<somename>")*
*     val input =
sc.textFile("hdfs://<servername>/user/cool/inputWrite.txt")*
*       input.count()*
*  }*
*}*

*Also see below the the .sbt file content. *

*name := "Simple Project"*

*version := "1.0"*

*scalaVersion := "2.9.3"*

*libraryDependencies ++=Seq("org.apache.spark" %% "spark-core" %
"0.8.0-incubating","org.apache.hadoop" % "hadoop-client" %
"2.0.4-alpha","com.google.protobuf" % "protobuf-java" % "2.4.1" force())*

*resolvers += "Akka Repository" at "http://repo.akka.io/releases/
<http://repo.akka.io/releases/>"*

*When i do "sbt run", I'm seeing below error. Can someone help me resolve
this issue? *

*java.io.IOException: Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Message missing
required fields: callId, status; Host Details : local host is: ;
destination host is: ;*
*java.io.IOException: Failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Message missing
required fields: callId, status; Host Details : local host is: ;
destination host is: ;*
*        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:761)*
*        at org.apache.hadoop.ipc.Client.call(Client.java:1239)*
*        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)*
*        at $Proxy12.getFileInfo(Unknown Source)*
*        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
*        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)*
*        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)*
*        at java.lang.reflect.Method.invoke(Method.java:597)*
*        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)*
*        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)*
*        at $Proxy12.getFileInfo(Unknown Source)*
*        at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:630)*
*        at
org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1559)*
*        at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:811)*
*        at
org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1649)*
*        at
org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1595)*
*        at
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:207)*
*        at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:251)*
*        at
org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:70)*
*        at org.apache.spark.rdd.RDD.partitions(RDD.scala:199)*
*        at
org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:26)*
*        at org.apache.spark.rdd.RDD.partitions(RDD.scala:199)*
*        at org.apache.spark.SparkContext.runJob(SparkContext.scala:772)*
*        at org.apache.spark.rdd.RDD.count(RDD.scala:677)*
*        at Hbase$.main(Hbase.scala:7)*
*        at Hbase.main(Hbase.scala)*
*        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
*        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)*
*        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)*
*        at java.lang.reflect.Method.invoke(Method.java:597)*
*Caused by: com.google.protobuf.InvalidProtocolBufferException: Message
missing required fields: callId, status*
*        at
com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:81)*
*        at
org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto$Builder.buildParsed(RpcPayloadHeaderProtos.java:1094)*
*        at
org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto$Builder.access$1300(RpcPayloadHeaderProtos.java:1028)*
*        at
org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcPayloadHeaderProtos.java:986)*
*        at
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:946)*
*        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:844)*
*[trace] Stack trace suppressed: run last compile:run for the full output.*
*java.lang.RuntimeException: Nonzero exit code: 1*
*        at scala.sys.package$.error(package.scala:27)*
*[trace] Stack trace suppressed: run last compile:run for the full output.*
*[error] (compile:run) Nonzero exit code: 1*
*[error] Total time: 3 s, completed Jan 20, 2014 12:57:32 PM*


Regards,
SB

Mime
View raw message