kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Asaf Mesika <asaf.mes...@gmail.com>
Subject Re: Spikes in kafka bytes out (while bytes in remain the same)
Date Wed, 13 Apr 2016 03:31:06 GMT
Where exactly do you get the measurement from? Your broker? Do you have
only one? Your producer? Your spark job?
On Mon, 11 Apr 2016 at 23:54 Jorge Rodriguez <jorge@bloomreach.com> wrote:

> We are running a kafka cluster for our real-time pixel processing
> pipeline.  The data is produced from our pixel servers into kafka, and then
> consumed by a spark streaming application.  Based on this, I would expect
> that the bytes in vs bytes out should be roughly equal, as each message
> should be consumed once.
>
> Under normal operations, the bytes out is a little less than 2X the bytes
> in.  Does anyone know why this is?  We do use a replication factor of 2.
>
> Occasionally, we get a spike in Bytes out.  But bytes in remain the same
> (see image below).  This correlates with a significant delay in processing
> time in the spark streaming side.
>
> Below is a chart of kafka reported bytes out vs in.  The system level
> network metrics show the same information (transferred bytes spike).
>
> Could anyone provide some tips for debugging/getting to the bottom of this
> issue?
>
> Thanks,
> Jorge
>
> *Kafka reported Bytes in Per topic and for all topics vs Kafka bytes out:*
>
> [image: Inline image 1]
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message