kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Praveen <praveev...@gmail.com>
Subject Kafka Consumer Committing Offset Even After Re-Assignment
Date Thu, 07 Dec 2017 20:56:29 GMT
I have 4 consumers on 2 boxes (running two consumers each) and 16
partitions. Each consumer takes 4 partitions.

In Kafka 0.9.0.1, I'm noticing that even when a consumer is no longer
assigned the partition, it is able to commit offset to it.

*Box 1 Started*
t1 - Box 1, Consumer 1 - Owns 8 partitions
      Box 1, Consumer 2 - Owns 8 partitions

      Consumers start polling and are submitting tasks to a task pool for
processing.

*Box 2 Started*
t2 - Box 1, Consumer 1 - Owns 4 partitions
      Box 1, Consumer 2 - Owns 4 partitions
      Box 2, Consumer 1 - Owns 4 partitions
      Box 2, Consumer 2 - Owns 4 partitions

      Partition-1 is now reassigned to Box 2, Consumer 1.
      But Box 1, Consumer 1 already submitted some of the records for
processing when it owned the partition earlier.

t3 - Box 1, Consumer 1 - After the tasks finish executing, even tho it
longer owns the partition, it is still able to commit the offset

t4 - Box 2, Consumer 1 - Commits offsets as well, overwriting offset
committed by Box 1, Consumer 1.

Is this expected? Should I be using the ConsumerRebalanceListener to
prevent commits to partitions not owned by the consumer?

- Praveen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message