kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Murtaza Doctor <murt...@richrelevance.com>
Subject Re: Trouble with consuming Avro messages
Date Mon, 24 Sep 2012 19:29:57 GMT
Hi Frank,

I had a similar issue, which was due to the avro schema mismatch on the
producer and consumer end. We are now working on an avro schema repository
service to handle this issue. If you can share your code snippet and stack
trace I will be able to better judge the issue.

Regards,
murtaza

On 9/24/12 10:57 AM, "Frank Grimes" <frankgrimes97@yahoo.com> wrote:

>Hi All,
>
>I'm new to Kafka and am having trouble sending/receiving messages in Avro
>format.
>I have Kafka 0.7.1-incubating talking to a standalone Zookeeper 3.3.6
>installation.
>The String producer/consumer examples in the quick start guide are
>working fine so I believe my setup and config are correct.
>
>After getting that working, I decided to write an encoder/decoder for my
>custom Avro record structure.
>They use Avro's DataFileWriter/DataFileStream to encode/decode the
>Message payload.
>I've tested my encoder and decoder standalone and they are working fine.
>
>The problem I am having is that when consuming these messages through a
>Kafka client they no longer deserialize correctly.
>Specifically, I get a NoSuchElementException thrown in
>DataFileStream.next().
>Also, I noticed that the Message payload size in the producer does not
>match the payload size in the consumer.
>
>Producer payload size: 2404
>Consumer payload size: 2326
>
>I tried disabling Avro compression but the problem remains.
>
>Any help would be greatly appreciated...
>
>Thanks,
>
>Frank Grimes


Mime
View raw message