kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Florian Hussonnois <fhussonn...@gmail.com>
Subject Re: Consumer Client - How to simulate heartbeats ?
Date Mon, 18 Apr 2016 13:59:45 GMT
Yes, but the ConsumerRebalanceListener is optional and by the default
KafkaConsumer uses a NoOpConsumerRebalanceListener if no one is provided.

I think the seek() is already done internally when a consumer joins or
quits the group. I'm not sure this line is actually needed.

2016-04-18 15:31 GMT+02:00 Kamal C <kamaltarget@gmail.com>:

> When a new consumer joins to the group, it should start to read data
> from where the other consumer left.
>
> --Kamal
>
> On Mon, Apr 18, 2016 at 6:58 PM, Florian Hussonnois <fhussonnois@gmail.com
> >
> wrote:
>
> > Thank you very much, the example is really helpful.
> >
> > My last question is : Why is it necessay to seek the consumer offsets
> into
> > the onPartitionsAssigned method ?
> >
> >
> >
> https://github.com/omkreddy/kafka-examples/blob/master/consumer/src/main/java/kafka/examples/consumer/advanced/AdvancedConsumer.java#L120
> >
> > 2016-04-15 15:06 GMT+02:00 Kamal C <kamaltarget@gmail.com>:
> >
> > > Hi Florian,
> > >
> > > This may be helpful
> > >
> > >
> >
> https://github.com/omkreddy/kafka-examples/blob/master/consumer/src/main/java/kafka/examples/consumer/advanced/AdvancedConsumer.java
> > >
> > > --Kamal
> > >
> > > On Fri, Apr 15, 2016 at 2:57 AM, Jason Gustafson <jason@confluent.io>
> > > wrote:
> > >
> > > > Hi Florian,
> > > >
> > > > It's actually OK if processing takes longer than the heartbeat
> > interval,
> > > > but it does need to finish before the session timeout expires or the
> > > > consumer will be kicked out of the group (which typically is revealed
> > by
> > > > commit failures). If the problem is just that the consumer is
> handling
> > > too
> > > > many messages at once, then Kafka 0.10 has an option to tune the
> number
> > > of
> > > > messages returned from poll() (max.poll.records), which may be
> helpful.
> > > We
> > > > also have a pause/resume API which allows you to call poll() without
> > > > consuming any data. That's the best option at the moment for 0.9
> > > consumers.
> > > >
> > > > For what it's worth, we've considered several times adding a
> > heartbeat()
> > > > API, but the challenge is figuring out how to handle rebalancing.
> > > > Underneath the covers, we use heartbeats to find out when the group
> is
> > > > rebalancing, so a heartbeat() option would probably have to return a
> > flag
> > > > indicating whether a rebalance was needed. If the group has begun
> > > > rebalancing, then you would need to call poll() before the expiration
> > of
> > > > the session timeout so that the consumer can join the rebalance.
> > > > Alternatively, we could let heartbeat() complete the rebalance
> itself,
> > > but
> > > > then you'd have to be prepared to abort processing from the rebalance
> > > > callback. That's not really different from calling poll() after
> pausing
> > > > partitions though. The main problem in any case is that once a
> > rebalance
> > > > begins, you have the duration of the session timeout to stop
> processing
> > > and
> > > > join the rebalance. We're seeing this problem pop up pretty much
> > > everywhere
> > > > that the consumer is used, so we're trying to think of some better
> > > options
> > > > to handle it.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > > > On Thu, Apr 14, 2016 at 12:32 PM, Florian Hussonnois <
> > > > fhussonnois@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I have a use case where a message can take longer than '
> > > > > heartbeat.interval.ms' to be processed by my application. As I
> > > > understand
> > > > > the heartbeats of consumer are done while the poll method is
> invoked.
> > > > >
> > > > > I would like to instantiate a worker thread to process the messages
> > > but I
> > > > > need to wait for the messages completion before polling again.
> > > > >
> > > > > Is there a way to force the consumer to make an heartbeat without
> > > polling
> > > > > new messages ?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > --
> > > > > Florian HUSSONNOIS
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Florian HUSSONNOIS
> >
>



-- 
Florian HUSSONNOIS

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message