kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Garvit Sharma <eng.gar...@gmail.com>
Subject Re: Kafka Topic Partition Consumer Lags
Date Thu, 27 Jun 2019 03:59:34 GMT
I don't think that is the case. The lag is huge ~10^5 records.

On Thu, Jun 27, 2019 at 9:13 AM Srinath C <srinath.c@gmail.com> wrote:

> Ok Garvit I still don't see the image but based on these inputs you
> provided I'm thinking that the possible scenario could be that between two
> polls from the consumer:
> (a) the number of records added to the partitions already consumed in
> previous poll is 500 or more (max.poll.records)
> or
> (b) the size of records in the partitions already consumed in previous poll
> is around ~50Mb (fetch.max.bytes)
>
> Regards,
> Srinath.
>
>
> On Thu, Jun 27, 2019 at 7:41 AM Garvit Sharma <eng.garvit@gmail.com>
> wrote:
>
> > Hi Srinath,
> >
> > I have attached the image.
> >
> > The partitions belong to the same topic only. I have not explicitly set
> > max.partition.fetch.bytes or fetch.max.bytes or max.poll.records so it
> > should take the default values.
> >
> > Let me know.
> >
> > Thanks,
> >
> > On Thu, Jun 27, 2019 at 7:11 AM Srinath C <srinath.c@gmail.com> wrote:
> >
> >> Hi Garvit,
> >>
> >> Am unable to see the image you attached for some reason and am not able
> to
> >> see if the partitions are in the same topic or in different topics.
> >> Check if any of max.partition.fetch.bytes or fetch.max.bytes or
> >> max.poll.records configured in your consumer is causing the behaviour.
> >>
> >> Regards,
> >> Srinath
> >>
> >>
> >> On Thu, Jun 27, 2019 at 5:43 AM Peter Bukowinski <pmbuko@gmail.com>
> >> wrote:
> >>
> >> > Is there a correlation between the lagging partitions and the consumer
> >> > assigned to them?
> >> >
> >> > > On Jun 26, 2019, at 4:25 PM, Garvit Sharma <eng.garvit@gmail.com>
> >> wrote:
> >> > >
> >> > > Can anyone please help me with this.
> >> > >
> >> > > On Wed, Jun 26, 2019 at 8:56 PM Garvit Sharma <eng.garvit@gmail.com
> >
> >> > wrote:
> >> > >
> >> > >> Hey Steve,
> >> > >>
> >> > >> I have checked, count of messages on all the partitions are same.
> >> > >>
> >> > >> I am still exploring an approach using which the root cause could
> be
> >> > >> determined.
> >> > >>
> >> > >> Thanks,
> >> > >>
> >> > >> On Wed, Jun 26, 2019 at 8:07 PM Garvit Sharma <
> eng.garvit@gmail.com>
> >> > >> wrote:
> >> > >>
> >> > >>> I am not sure about that. Is there a way to analyse that ?
> >> > >>>
> >> > >>> On Wed, Jun 26, 2019 at 7:35 PM Steve Howard <
> >> > steve.howard@confluent.io>
> >> > >>> wrote:
> >> > >>>
> >> > >>>> Hi Garvit,
> >> > >>>>
> >> > >>>> Are the slow partitions "hot", i.e., receiving a lot more
> messages
> >> > than
> >> > >>>> others?
> >> > >>>>
> >> > >>>> Thanks,
> >> > >>>>
> >> > >>>> Steve
> >> > >>>>
> >> > >>>> On Wed, Jun 26, 2019, 9:56 AM Garvit Sharma <
> eng.garvit@gmail.com
> >> > wrote:
> >> > >>>>
> >> > >>>>> Just to add more details, these consumers are processing
the
> Kafka
> >> > >>>> events
> >> > >>>>> and writing to DB(fast write guaranteed).
> >> > >>>>>
> >> > >>>>> On Wed, Jun 26, 2019 at 7:23 PM Garvit Sharma <
> >> eng.garvit@gmail.com>
> >> > >>>>> wrote:
> >> > >>>>>
> >> > >>>>>> Hi All,
> >> > >>>>>>
> >> > >>>>>> I can see huge consumer lag in a few partitions
of Kafka
> topic. I
> >> > >>>> need to
> >> > >>>>>> know the root cause of this issue.
> >> > >>>>>>
> >> > >>>>>> Please let me know, how to proceed.
> >> > >>>>>>
> >> > >>>>>> Below is sample consumer lag data :
> >> > >>>>>>
> >> > >>>>>> [image: image.png]
> >> > >>>>>>
> >> > >>>>>> Thanks,
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>
> >> > >>>>
> >> > >>>
> >> >
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message