kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kamal C <kamaltar...@gmail.com>
Subject Re: Consumer Client - How to simulate heartbeats ?
Date Fri, 15 Apr 2016 13:06:58 GMT
Hi Florian,

This may be helpful
https://github.com/omkreddy/kafka-examples/blob/master/consumer/src/main/java/kafka/examples/consumer/advanced/AdvancedConsumer.java

--Kamal

On Fri, Apr 15, 2016 at 2:57 AM, Jason Gustafson <jason@confluent.io> wrote:

> Hi Florian,
>
> It's actually OK if processing takes longer than the heartbeat interval,
> but it does need to finish before the session timeout expires or the
> consumer will be kicked out of the group (which typically is revealed by
> commit failures). If the problem is just that the consumer is handling too
> many messages at once, then Kafka 0.10 has an option to tune the number of
> messages returned from poll() (max.poll.records), which may be helpful. We
> also have a pause/resume API which allows you to call poll() without
> consuming any data. That's the best option at the moment for 0.9 consumers.
>
> For what it's worth, we've considered several times adding a heartbeat()
> API, but the challenge is figuring out how to handle rebalancing.
> Underneath the covers, we use heartbeats to find out when the group is
> rebalancing, so a heartbeat() option would probably have to return a flag
> indicating whether a rebalance was needed. If the group has begun
> rebalancing, then you would need to call poll() before the expiration of
> the session timeout so that the consumer can join the rebalance.
> Alternatively, we could let heartbeat() complete the rebalance itself, but
> then you'd have to be prepared to abort processing from the rebalance
> callback. That's not really different from calling poll() after pausing
> partitions though. The main problem in any case is that once a rebalance
> begins, you have the duration of the session timeout to stop processing and
> join the rebalance. We're seeing this problem pop up pretty much everywhere
> that the consumer is used, so we're trying to think of some better options
> to handle it.
>
> Thanks,
> Jason
>
>
> On Thu, Apr 14, 2016 at 12:32 PM, Florian Hussonnois <
> fhussonnois@gmail.com>
> wrote:
>
> > Hi everyone,
> >
> > I have a use case where a message can take longer than '
> > heartbeat.interval.ms' to be processed by my application. As I
> understand
> > the heartbeats of consumer are done while the poll method is invoked.
> >
> > I would like to instantiate a worker thread to process the messages but I
> > need to wait for the messages completion before polling again.
> >
> > Is there a way to force the consumer to make an heartbeat without polling
> > new messages ?
> >
> > Thanks,
> >
> > --
> > Florian HUSSONNOIS
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message