kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Data loss detection
Date Wed, 04 Jun 2014 20:26:04 GMT
It should be sth like clientId-MessagesPerSec.

Thanks,

Jun


On Wed, Jun 4, 2014 at 9:35 AM, Maung Than <maung_than@apple.com> wrote:

>
> We could not find producer msg rate from the matrices in the JConsole —
> give us some pointers.
>
> Also confirming that the reduction in data is due to Avro encoding and we
> are calculating what we send to producer rather than the output of
> serializer encoder.
>
> Thanks,
> Maung
>
> On Jun 3, 2014, at 10:47 PM, Maung Than <maung_than@apple.com> wrote:
>
> > Thanks, Jun.
> >
> > Will check and get back..
> >
> > We are converting JSON to Avro and that conversion is done by the custom
> serializer.
> >
> > Our volume calculation on the producer side is based on the AVRO generic
> record that is passed to the producer send method— not of the encoded data
> output from the serializer that is what actual got send the Broker I
> believe.
> >
> > That could be the gap and I am testing now without the customer
> serializer and seeing the two volumes are very close.  That could be it!!
> >
> > Thanks,
> > Maung
> >
> > On Jun 3, 2014, at 7:22 PM, Jun Rao <junrao@gmail.com> wrote:
> >
> >> We have a metric on msg rate in both the producer and the broker. Could
> you
> >> see if they match?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Tue, Jun 3, 2014 at 2:13 PM, Maung Than <maung_than@apple.com>
> wrote:
> >>
> >>> Hi,
> >>>
> >>> We are seeing less data on the brokers than we send form the producers:
> >>> 84 GB to 58 GB.
> >>>
> >>> What is the best way to ensure / detect if all data has been send
> properly
> >>> to the brokers from the producers.
> >>>
> >>> Is there any logs that we can check on the producers?
> >>>
> >>> Configuration is 5 Brokers, 2 producers, no replication factor, async
> and
> >>> ask is 1 and no compression.
> >>>
> >>> Thanks,
> >>> Maung
> >>>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message