kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dennis Haller <dhal...@talemetry.com>
Subject How does Kafka decide which Consumer out of multiple Consumer clients to assign to a single topic partition
Date Thu, 22 Aug 2013 15:53:46 GMT
I have a situation where two high level consumers are being created to
consume a single topic. There is only one partition for the topic, so I
understand that only one Consumer will end up owning the topic and
receiving messages. The two consumers are created from two servers in a
redundant master - slave configuration, and it is our intention that the
servers should start in the same configuration predictably, with all the
Consumers active on the master server.

However, we find after both Consumers have been created that sometimes the
first Consumer client succeeds in owning the topic and sometimes it is the
second. I first thought that the first Consumer  client to register with
the topic would be retained even if subsequent Consumers also register for
the topic, but I see sometimes the second Consumer client succeeds in
replacing the first Consumer.

The logs show a rebalancing algorithm working after each Consumer is
registered.

In this case, where there is only one topic-partition, is it possible to
predict what Consumer client will own the topic? How is that rebalancing
done?

Thanks
Dennis

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message