kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jorge Rodriguez <jo...@bloomreach.com>
Subject Re: Spikes in kafka bytes out (while bytes in remain the same)
Date Wed, 20 Apr 2016 19:57:09 GMT
Asaf, thanks for your explanation.  This actually makes complete sense, as
we have 2 replicas.  So the math works out when taking this into
consideration.

Thanks!
Jorge

On Sat, Apr 16, 2016 at 9:32 PM, Asaf Mesika <asaf.mesika@gmail.com> wrote:

> Another thought: Brokers replicate data in. So a record weighing 10 bytes
> will be written out once for replication and one more time to a consumer so
> it will be 20 bytes out. Makes sense?
> On Thu, 14 Apr 2016 at 02:46 Jorge Rodriguez <jorge@bloomreach.com> wrote:
>
> > Thanks for your response Asaf.  I have 4 brokers.  These measurements are
> > from the kafka brokers.
> >
> > This measurement on this graph comes from Kafka.  It is a sum across all
> 4
> > brokers of the
> > metric: kafka.server.BrokerTopicMetrics.BytesInPerSec.1MinuteRate.
> >
> > But I also have a system metric which I feed independently using collectd
> > "interface" plugin.  And the bytes out and in match the ones reported by
> > kafka fairly well.  As well there is a corresponding increase in network
> > packets sent.
> >
> > Also, in the SparkStreaming side, I can see that during these spikes, the
> > number of received packets and bytes also spikes.
> >
> > So during the spikes, I believe that some of the fetch requests are
> perhaps
> > failing and we hit a retry.  I am debugging that currently and I think
> it's
> > related to the STW GC which happens on spark streaming occasionally.
> > Working on some GC tuning should alleviate this.
> >
> > However, even if this is the case, this would not explain though why
> under
> > normal operations, the number of bytes out is 2x the number of bytes in.
> > Since I only have 1 consumer for each topic, I would expect the numbers
> to
> > be fairly close.  Do you
> >
> >
> >
> >
> > On Tue, Apr 12, 2016 at 8:31 PM, Asaf Mesika <asaf.mesika@gmail.com>
> > wrote:
> >
> > > Where exactly do you get the measurement from? Your broker? Do you have
> > > only one? Your producer? Your spark job?
> > > On Mon, 11 Apr 2016 at 23:54 Jorge Rodriguez <jorge@bloomreach.com>
> > wrote:
> > >
> > > > We are running a kafka cluster for our real-time pixel processing
> > > > pipeline.  The data is produced from our pixel servers into kafka,
> and
> > > then
> > > > consumed by a spark streaming application.  Based on this, I would
> > expect
> > > > that the bytes in vs bytes out should be roughly equal, as each
> message
> > > > should be consumed once.
> > > >
> > > > Under normal operations, the bytes out is a little less than 2X the
> > bytes
> > > > in.  Does anyone know why this is?  We do use a replication factor of
> > 2.
> > > >
> > > > Occasionally, we get a spike in Bytes out.  But bytes in remain the
> > same
> > > > (see image below).  This correlates with a significant delay in
> > > processing
> > > > time in the spark streaming side.
> > > >
> > > > Below is a chart of kafka reported bytes out vs in.  The system level
> > > > network metrics show the same information (transferred bytes spike).
> > > >
> > > > Could anyone provide some tips for debugging/getting to the bottom of
> > > this
> > > > issue?
> > > >
> > > > Thanks,
> > > > Jorge
> > > >
> > > > *Kafka reported Bytes in Per topic and for all topics vs Kafka bytes
> > > out:*
> > > >
> > > > [image: Inline image 1]
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message