kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saïd Bouras <said.bou...@gmail.com>
Subject Re: Kafka Consumer Committing Offset Even After Re-Assignment
Date Fri, 08 Dec 2017 10:14:19 GMT
Hi Praveen,

I don't know if you that's your case but if you know that consumers will
lost ownership of partitions, you have to use the
*ConsumerRebalanceListener* to drop the last offset of record processed in
a clean way.

If you don't do that, the rebalance will start when the GroupCoordinator
stopped to receive heartbeats from the Consumer.
I don't know if that will solve your problem, but the consumer after
commiting his last offset will leave the ConsumerGroup.

Regards


On Thu, Dec 7, 2017 at 9:58 PM Praveen <praveev.gk@gmail.com> wrote:

> I have 4 consumers on 2 boxes (running two consumers each) and 16
> partitions. Each consumer takes 4 partitions.
>
> In Kafka 0.9.0.1, I'm noticing that even when a consumer is no longer
> assigned the partition, it is able to commit offset to it.
>
> *Box 1 Started*
> t1 - Box 1, Consumer 1 - Owns 8 partitions
>       Box 1, Consumer 2 - Owns 8 partitions
>
>       Consumers start polling and are submitting tasks to a task pool for
> processing.
>
> *Box 2 Started*
> t2 - Box 1, Consumer 1 - Owns 4 partitions
>       Box 1, Consumer 2 - Owns 4 partitions
>       Box 2, Consumer 1 - Owns 4 partitions
>       Box 2, Consumer 2 - Owns 4 partitions
>
>       Partition-1 is now reassigned to Box 2, Consumer 1.
>       But Box 1, Consumer 1 already submitted some of the records for
> processing when it owned the partition earlier.
>
> t3 - Box 1, Consumer 1 - After the tasks finish executing, even tho it
> longer owns the partition, it is still able to commit the offset
>
> t4 - Box 2, Consumer 1 - Commits offsets as well, overwriting offset
> committed by Box 1, Consumer 1.
>
> Is this expected? Should I be using the ConsumerRebalanceListener to
> prevent commits to partitions not owned by the consumer?
>
> - Praveen
>
-- 

Saïd BOURAS

Consultant Big Data
Mobile: 0662988731
Zenika Paris
10 rue de Milan 75009 Paris
Standard : +33(0)1 45 26 19 15 <+33(0)145261915> - Fax : +33(0)1 72 70 45 10
<+33(0)172704510>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message