kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Grimes <frankgrime...@yahoo.com>
Subject Trouble with consuming Avro messages
Date Mon, 24 Sep 2012 17:57:44 GMT
Hi All,

I'm new to Kafka and am having trouble sending/receiving messages in Avro format.
I have Kafka 0.7.1-incubating talking to a standalone Zookeeper 3.3.6 installation.
The String producer/consumer examples in the quick start guide are working fine so I believe
my setup and config are correct.

After getting that working, I decided to write an encoder/decoder for my custom Avro record
structure.
They use Avro's DataFileWriter/DataFileStream to encode/decode the Message payload.
I've tested my encoder and decoder standalone and they are working fine.

The problem I am having is that when consuming these messages through a Kafka client they
no longer deserialize correctly.
Specifically, I get a NoSuchElementException thrown in DataFileStream.next().
Also, I noticed that the Message payload size in the producer does not match the payload size
in the consumer.

Producer payload size: 2404
Consumer payload size: 2326

I tried disabling Avro compression but the problem remains.

Any help would be greatly appreciated...

Thanks,

Frank Grimes
Mime
View raw message