kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rodenburg, Jeff" <jeff.rodenb...@teamaol.com>
Subject Re: Consumer group concept
Date Tue, 12 Jun 2012 18:34:10 GMT
Thanks for the info, Jun.

>  if you just want each message to be consumed by a consumer, not a particular one

What is intended to be a particular consumer? Something on the order of Consumer #3 within
a group needs message #123?

Ok, next question:

What is the relationship between topics and consumer groups? More to the point, can I have
multiple consumer groups that all consume the same topic?  For example, assume a set of producers
are publishing to the topic "ABC".  Suppose I have multiple processes that take action on
a given ABC message -- process 1 handles billing, process 2 handles file management, process
3 handles history/archiving, etc.  Can I structure multiple groups that consume the same topic?
How does partitioning work at that point?




On Jun 12, 2012, at 10:11 AM, Jun Rao wrote:

> Jeff,
> 
> Your understanding is correct. Operational wise, we have some jmx that
> gives consumer stats per topic. There is also a tool CheckOffsetLag that
> tells you how far behind a consumer is. For coordination btw producers and
> consumers, if you just want each message to be consumed by a consumer, not
> a particular one, there is no coordination needed.
> 
> Thanks,
> 
> Jun
> 
> On Tue, Jun 12, 2012 at 9:58 AM, Rodenburg, Jeff <jeff.rodenburg@teamaol.com
>> wrote:
> 
>> Hi all -
>> 
>> Just getting familiar with Kafka, and learning about consumer groups.
>> Hoping someone can provide some context here.
>> 
>> As I understand it, consumers register with the broker and consume a
>> topic. Multiple consumers can consume a single topic, as a consumer group.
>> Each consumer actually gets a partition of messages, so there is no overlap
>> -- a single consumer within a group will receive a message on its
>> topic/partition.  Consumer rebalancing is the process whereby members of a
>> consumer group are added and/or dropped from the group, and partitions are
>> sorted/reassigned to the current consumer group members.
>> 
>> Some questions:
>> 
>> *   Is this accurate? What am I missing?
>> *   Operationally, is consumer "failover" basically service monitoring at
>> the consumer process level?
>> *   How much coordination is required between producers and consumers
>> around partitioning? (Automated, configuration, etc.)
>> *   How are topics monitored for SLA on throughput/load, i.e. spinning up
>> consumers as needed for topic message spikes?
>> 
>> Appreciate any further information and/or context anyone can share.
>> 
>> cheers,
>> Jeff
>> 


Mime
View raw message