kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maung Than <maung_t...@apple.com>
Subject Re: Data loss detection
Date Wed, 04 Jun 2014 05:47:51 GMT
Thanks, Jun. 

Will check and get back..

We are converting JSON to Avro and that conversion is done by the custom serializer. 

Our volume calculation on the producer side is based on the AVRO generic record that is passed
to the producer send method— not of the encoded data output from the serializer that is
what actual got send the Broker I believe. 

That could be the gap and I am testing now without the customer serializer and seeing the
two volumes are very close.  That could be it!!

Thanks,
Maung

On Jun 3, 2014, at 7:22 PM, Jun Rao <junrao@gmail.com> wrote:

> We have a metric on msg rate in both the producer and the broker. Could you
> see if they match?
> 
> Thanks,
> 
> Jun
> 
> 
> On Tue, Jun 3, 2014 at 2:13 PM, Maung Than <maung_than@apple.com> wrote:
> 
>> Hi,
>> 
>> We are seeing less data on the brokers than we send form the producers:
>> 84 GB to 58 GB.
>> 
>> What is the best way to ensure / detect if all data has been send properly
>> to the brokers from the producers.
>> 
>> Is there any logs that we can check on the producers?
>> 
>> Configuration is 5 Brokers, 2 producers, no replication factor, async and
>> ask is 1 and no compression.
>> 
>> Thanks,
>> Maung
>> 


Mime
View raw message