kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Loddengaard <a...@confluent.io>
Subject Re: Consumer stopped after reading some messages
Date Tue, 10 May 2016 22:44:21 GMT
Hi Sahitya,

I wonder if your consumers are experiencing soft failures because they're
busy processing a large collection of messages and not calling poll()
within session.timeout.ms? In this scenario, the group coordinator (a
broker) would not receive a heartbeat within session.timeout.ms and would
consider the consumer failed. The coordinator would then reassign the
"failed" consumer's partitions to other consumers in the same group. If all
consumers are experiencing soft failures, you may observe them all
"freezing" their consumption. I suggest checking the logs to see if your
consumer group is being rebalanced frequently.

If you are hitting the issue I've explained, checkout this KIP:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records

Or consider increasing session.timeout.ms.

Hope this helps.

Alex

On Thu, May 5, 2016 at 8:22 AM, John Bickerstaff <john@johnbickerstaff.com>
wrote:

> This may or may not help.  I found it to be a clever workaround for some of
> the limitations in the 8.x version of the high level consumer...  I ended
> up writing code that "waited" a lot because I couldn't be sure how quickly
> Kafka would respond...
>
> Nothing ever took minutes however...  the waits were 30 seconds or so if I
> recall...
>
> In case it helps...
>
> http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-
> pieces/
> <http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/>
>
> On Thu, May 5, 2016 at 4:58 AM, sahitya agrawal <sahitya2109@gmail.com>
> wrote:
>
> > I am using high level consumer API ( Kafka API version 0.9.0.0 )
> >
> > I am running consumers on a topic of 10 partitions. There are lot of
> unread
> > messages in that topic. Initially all of them , consume from the topic
> and
> > read messages. After sometime, all of them hangs and doesn't read any
> > message at all.
> > I have to manually restart them to make them consume again.
> > Surprisingly I do not see any exception or error in logs also.
> >
> > Has anybody ever faced this issue?
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message