kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Grimes <frankgrime...@yahoo.com>
Subject Re: Trouble with consuming Avro messages
Date Mon, 24 Sep 2012 19:44:36 GMT
Hi Murtaza,

I don't think it's the same issue.
I'm using the exact same schema on both ends. 
I generated an IndexedRecord from an avsc file and both the encoder and decoder reference
the same data type (and therefore the same IndexedRecord.$SCHEMA).

Also, the payload size mismatch is on the raw ByteBuffer before decoding occurs, is it not?
(No Avro/schema involved on the decoding side yet)

Regards,

Emanuel


On 2012-09-24, at 3:29 PM, Murtaza Doctor <murtaza@richrelevance.com> wrote:

> Hi Frank,
> 
> I had a similar issue, which was due to the avro schema mismatch on the
> producer and consumer end. We are now working on an avro schema repository
> service to handle this issue. If you can share your code snippet and stack
> trace I will be able to better judge the issue.
> 
> Regards,
> murtaza
> 
> On 9/24/12 10:57 AM, "Frank Grimes" <frankgrimes97@yahoo.com> wrote:
> 
>> Hi All,
>> 
>> I'm new to Kafka and am having trouble sending/receiving messages in Avro
>> format.
>> I have Kafka 0.7.1-incubating talking to a standalone Zookeeper 3.3.6
>> installation.
>> The String producer/consumer examples in the quick start guide are
>> working fine so I believe my setup and config are correct.
>> 
>> After getting that working, I decided to write an encoder/decoder for my
>> custom Avro record structure.
>> They use Avro's DataFileWriter/DataFileStream to encode/decode the
>> Message payload.
>> I've tested my encoder and decoder standalone and they are working fine.
>> 
>> The problem I am having is that when consuming these messages through a
>> Kafka client they no longer deserialize correctly.
>> Specifically, I get a NoSuchElementException thrown in
>> DataFileStream.next().
>> Also, I noticed that the Message payload size in the producer does not
>> match the payload size in the consumer.
>> 
>> Producer payload size: 2404
>> Consumer payload size: 2326
>> 
>> I tried disabling Avro compression but the problem remains.
>> 
>> Any help would be greatly appreciated...
>> 
>> Thanks,
>> 
>> Frank Grimes
> 


Mime
View raw message