kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: mBean to monitor message per partitions in topic
Date Fri, 30 May 2014 03:34:15 GMT
If you open up jmx (e.g. jconsole) in a consumer instance, you will see the
jmx of name *-ConsumerLag.

Thanks,

Jun


On Thu, May 29, 2014 at 8:53 AM, Рябков Алексей Николаевич <
a.ryabkov@ntc-vulkan.ru> wrote:

> Do you mean getOffsetLag ? (
> https://cwiki.apache.org/confluence/display/KAFKA/Operations)
>
> But its all about offset not message...
>
> So for example to create "real" load balance between broker I can:
>
> 1. Calculate average unread message in topic per broker:
>        - Sall=sum(kafka.BrokerAllTopicStat.[topic]. getMessagesIn) -total
> send messages in topic for all broker
>        - Rall=sum (kafka.ConsumerTopicStat.[topic]. getMessagesPerTopic -
>  -total read messages in topic for all consumers
>        - (Sall-Rall)/broker number - average unread messages in topic per
> broker
> 2. Get current number of unread messages in topic for broker (little hack
> here)
>       for example if we  locate N consumers per broker and each consumer
> must read only from 1 partition than we can :
>        - Sb=kafka.BrokerAllTopicStat.[topic]. getMessagesIn) -total send
> messages in topic for broker B
>        - Rb=sum for all N consumers in broker b
> (kafka.ConsumerTopicStat.[topic]. getMessagesPerTopic) - total unread
> messages in topic for N consumers on broker b
>         - (Sb-Rb) -unread messages in topic for broker
> 3. Use partitioner to load balance between brokers ...
>         - compare average unread messages in topic per broker with current
> unread messages in topic for broker we can create more clever load
> balancer...
>
>
> But what can you tell about performance?... How fast I can get monitoring
> stat?  Could you give me some advice for optimization?
>
>
> Thanks, Aleksey Ryabkov
>
>
> -----Исходное сообщение-----
> От: Jun Rao [mailto:junrao@gmail.com]
> Отправлено: Thursday, May 29, 2014 8:34 AM
> Кому: users@kafka.apache.org
> Тема: Re: mBean to monitor message per partitions in topic
>
> There is a per-partition jmx (*-ConsumerLag) in the consumer that reports
> unconsumed messages per partition.
>
> Thanks,
>
> Jun
>
>
> On Wed, May 28, 2014 at 8:13 AM, Рябков Алексей Николаевич <
> a.ryabkov@ntc-vulkan.ru> wrote:
>
> > Hello!
> >
> > How can I get information about unfetched message per partition in topic?
> >        I wish to use such information to create my custom
> > partitioner.class to balance messages between partitions
> >
> > With best regards, Aleksey Ryabkov
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message