kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Consumer group concept
Date Tue, 12 Jun 2012 17:11:38 GMT
Jeff,

Your understanding is correct. Operational wise, we have some jmx that
gives consumer stats per topic. There is also a tool CheckOffsetLag that
tells you how far behind a consumer is. For coordination btw producers and
consumers, if you just want each message to be consumed by a consumer, not
a particular one, there is no coordination needed.

Thanks,

Jun

On Tue, Jun 12, 2012 at 9:58 AM, Rodenburg, Jeff <jeff.rodenburg@teamaol.com
> wrote:

> Hi all -
>
> Just getting familiar with Kafka, and learning about consumer groups.
> Hoping someone can provide some context here.
>
> As I understand it, consumers register with the broker and consume a
> topic. Multiple consumers can consume a single topic, as a consumer group.
> Each consumer actually gets a partition of messages, so there is no overlap
> -- a single consumer within a group will receive a message on its
> topic/partition.  Consumer rebalancing is the process whereby members of a
> consumer group are added and/or dropped from the group, and partitions are
> sorted/reassigned to the current consumer group members.
>
> Some questions:
>
>  *   Is this accurate? What am I missing?
>  *   Operationally, is consumer "failover" basically service monitoring at
> the consumer process level?
>  *   How much coordination is required between producers and consumers
> around partitioning? (Automated, configuration, etc.)
>  *   How are topics monitored for SLA on throughput/load, i.e. spinning up
> consumers as needed for topic message spikes?
>
> Appreciate any further information and/or context anyone can share.
>
> cheers,
> Jeff
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message