kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Gustafson <ja...@confluent.io>
Subject Re: Consumer Client - How to simulate heartbeats ?
Date Thu, 14 Apr 2016 21:27:34 GMT
Hi Florian,

It's actually OK if processing takes longer than the heartbeat interval,
but it does need to finish before the session timeout expires or the
consumer will be kicked out of the group (which typically is revealed by
commit failures). If the problem is just that the consumer is handling too
many messages at once, then Kafka 0.10 has an option to tune the number of
messages returned from poll() (max.poll.records), which may be helpful. We
also have a pause/resume API which allows you to call poll() without
consuming any data. That's the best option at the moment for 0.9 consumers.

For what it's worth, we've considered several times adding a heartbeat()
API, but the challenge is figuring out how to handle rebalancing.
Underneath the covers, we use heartbeats to find out when the group is
rebalancing, so a heartbeat() option would probably have to return a flag
indicating whether a rebalance was needed. If the group has begun
rebalancing, then you would need to call poll() before the expiration of
the session timeout so that the consumer can join the rebalance.
Alternatively, we could let heartbeat() complete the rebalance itself, but
then you'd have to be prepared to abort processing from the rebalance
callback. That's not really different from calling poll() after pausing
partitions though. The main problem in any case is that once a rebalance
begins, you have the duration of the session timeout to stop processing and
join the rebalance. We're seeing this problem pop up pretty much everywhere
that the consumer is used, so we're trying to think of some better options
to handle it.

Thanks,
Jason


On Thu, Apr 14, 2016 at 12:32 PM, Florian Hussonnois <fhussonnois@gmail.com>
wrote:

> Hi everyone,
>
> I have a use case where a message can take longer than '
> heartbeat.interval.ms' to be processed by my application. As I understand
> the heartbeats of consumer are done while the poll method is invoked.
>
> I would like to instantiate a worker thread to process the messages but I
> need to wait for the messages completion before polling again.
>
> Is there a way to force the consumer to make an heartbeat without polling
> new messages ?
>
> Thanks,
>
> --
> Florian HUSSONNOIS
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message