kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maung Than <maung_t...@apple.com>
Subject Re: Data loss detection
Date Wed, 04 Jun 2014 16:35:02 GMT

We could not find producer msg rate from the matrices in the JConsole — give us some pointers.


Also confirming that the reduction in data is due to Avro encoding and we are calculating
what we send to producer rather than the output of serializer encoder. 

Thanks,
Maung

On Jun 3, 2014, at 10:47 PM, Maung Than <maung_than@apple.com> wrote:

> Thanks, Jun. 
> 
> Will check and get back..
> 
> We are converting JSON to Avro and that conversion is done by the custom serializer.

> 
> Our volume calculation on the producer side is based on the AVRO generic record that
is passed to the producer send method— not of the encoded data output from the serializer
that is what actual got send the Broker I believe. 
> 
> That could be the gap and I am testing now without the customer serializer and seeing
the two volumes are very close.  That could be it!!
> 
> Thanks,
> Maung
> 
> On Jun 3, 2014, at 7:22 PM, Jun Rao <junrao@gmail.com> wrote:
> 
>> We have a metric on msg rate in both the producer and the broker. Could you
>> see if they match?
>> 
>> Thanks,
>> 
>> Jun
>> 
>> 
>> On Tue, Jun 3, 2014 at 2:13 PM, Maung Than <maung_than@apple.com> wrote:
>> 
>>> Hi,
>>> 
>>> We are seeing less data on the brokers than we send form the producers:
>>> 84 GB to 58 GB.
>>> 
>>> What is the best way to ensure / detect if all data has been send properly
>>> to the brokers from the producers.
>>> 
>>> Is there any logs that we can check on the producers?
>>> 
>>> Configuration is 5 Brokers, 2 producers, no replication factor, async and
>>> ask is 1 and no compression.
>>> 
>>> Thanks,
>>> Maung
>>> 
> 


Mime
View raw message