kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rodenburg, Jeff" <jeff.rodenb...@teamaol.com>
Subject Consumer group concept
Date Tue, 12 Jun 2012 16:58:15 GMT
Hi all -

Just getting familiar with Kafka, and learning about consumer groups. Hoping someone can provide
some context here.

As I understand it, consumers register with the broker and consume a topic. Multiple consumers
can consume a single topic, as a consumer group. Each consumer actually gets a partition of
messages, so there is no overlap -- a single consumer within a group will receive a message
on its topic/partition.  Consumer rebalancing is the process whereby members of a consumer
group are added and/or dropped from the group, and partitions are sorted/reassigned to the
current consumer group members.

Some questions:

  *   Is this accurate? What am I missing?
  *   Operationally, is consumer "failover" basically service monitoring at the consumer process
  *   How much coordination is required between producers and consumers around partitioning?
(Automated, configuration, etc.)
  *   How are topics monitored for SLA on throughput/load, i.e. spinning up consumers as needed
for topic message spikes?

Appreciate any further information and/or context anyone can share.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message