kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srinath C <srinat...@gmail.com>
Subject Re: Kafka Topic Partition Consumer Lags
Date Thu, 27 Jun 2019 01:41:22 GMT
Hi Garvit,

Am unable to see the image you attached for some reason and am not able to
see if the partitions are in the same topic or in different topics.
Check if any of max.partition.fetch.bytes or fetch.max.bytes or
max.poll.records configured in your consumer is causing the behaviour.

Regards,
Srinath


On Thu, Jun 27, 2019 at 5:43 AM Peter Bukowinski <pmbuko@gmail.com> wrote:

> Is there a correlation between the lagging partitions and the consumer
> assigned to them?
>
> > On Jun 26, 2019, at 4:25 PM, Garvit Sharma <eng.garvit@gmail.com> wrote:
> >
> > Can anyone please help me with this.
> >
> > On Wed, Jun 26, 2019 at 8:56 PM Garvit Sharma <eng.garvit@gmail.com>
> wrote:
> >
> >> Hey Steve,
> >>
> >> I have checked, count of messages on all the partitions are same.
> >>
> >> I am still exploring an approach using which the root cause could be
> >> determined.
> >>
> >> Thanks,
> >>
> >> On Wed, Jun 26, 2019 at 8:07 PM Garvit Sharma <eng.garvit@gmail.com>
> >> wrote:
> >>
> >>> I am not sure about that. Is there a way to analyse that ?
> >>>
> >>> On Wed, Jun 26, 2019 at 7:35 PM Steve Howard <
> steve.howard@confluent.io>
> >>> wrote:
> >>>
> >>>> Hi Garvit,
> >>>>
> >>>> Are the slow partitions "hot", i.e., receiving a lot more messages
> than
> >>>> others?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Steve
> >>>>
> >>>> On Wed, Jun 26, 2019, 9:56 AM Garvit Sharma <eng.garvit@gmail.com
> wrote:
> >>>>
> >>>>> Just to add more details, these consumers are processing the Kafka
> >>>> events
> >>>>> and writing to DB(fast write guaranteed).
> >>>>>
> >>>>> On Wed, Jun 26, 2019 at 7:23 PM Garvit Sharma <eng.garvit@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi All,
> >>>>>>
> >>>>>> I can see huge consumer lag in a few partitions of Kafka topic.
I
> >>>> need to
> >>>>>> know the root cause of this issue.
> >>>>>>
> >>>>>> Please let me know, how to proceed.
> >>>>>>
> >>>>>> Below is sample consumer lag data :
> >>>>>>
> >>>>>> [image: image.png]
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message