spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cassa L <lcas...@gmail.com>
Subject Re: Protobuf error when streaming from Kafka
Date Tue, 25 Aug 2015 19:39:46 GMT
Hi,
 I am using Spark-1.4 and Kafka-0.8.2.1
As per google suggestions, I rebuilt all the classes with protobuff-2.5
dependencies. My new protobuf is compiled using 2.5. However now, my spark
job does not start. Its throwing different error. Does Spark or any other
its dependencies uses old protobuff-2.4?

Exception in thread "main" java.lang.VerifyError: class
com.apple.ist.retail.xcardmq.serializers.SampleProtobufMessage$ProtoBuff
overrides final method
getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at
com.apple.ist.retail.spark.kafka.consumer.SparkMQProcessor.start(SparkProcessor.java:68)
    at
com.apple.ist.retail.spark.kafka.consumer.SparkMQConsumer.main(SparkConsumer.java:43)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
    at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)


On Mon, Aug 24, 2015 at 6:53 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Can you show the complete stack trace ?
>
> Which Spark / Kafka release are you using ?
>
> Thanks
>
> On Mon, Aug 24, 2015 at 4:58 PM, Cassa L <lcassa8@gmail.com> wrote:
>
>> Hi,
>>  I am storing messages in Kafka using protobuf and reading them into
>> Spark. I upgraded protobuf version from 2.4.1 to 2.5.0. I got
>> "java.lang.UnsupportedOperationException" for older messages. However, even
>> for new messages I get the same error. Spark does convert it though. I see
>> my messages. How do I get rid of this error?
>> java.lang.UnsupportedOperationException: This is supposed to be
>> overridden by subclasses.
>>     at
>> com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
>>     at
>> org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$FsPermissionProto.getSerializedSize(HdfsProtos.java:5407)
>>     at
>> com.google.protobuf.CodedOutputStream.computeMessageSizeNoTag(CodedOutputStream.java:749)
>>
>
>

Mime
View raw message