kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marina <ppi...@yahoo.com.INVALID>
Subject Re: Kafka JMS metrics meaning
Date Tue, 02 Jun 2015 17:42:36 GMT
Thanks a lot to everybody for your suggestions! 
In addition to the Consumer lag (on the Consumers side though), under-replicated partitions,
offline partitions, active controller count, I am also thinking of monitoring the total size
of partitions to not exceed some MAX (like 10G, for example) - to prevent disk out of space
issues.

Now, can somebody shed light on my second question :) :"2. what do these metrics mean: ReplicaManager
-> LeaderCount ? and ReplicaManager -> PartitionCount ?I have three topics created,
with one partition each, and replication = 1, however the values for both of the above attributes
is "53".... So I am not sure what the count '53' means here...."
thanks!MArina

      From: Todd Palino <tpalino@gmail.com>
 To: "users@kafka.apache.org" <users@kafka.apache.org> 
Cc: Marina <ppine7@yahoo.com> 
 Sent: Tuesday, June 2, 2015 1:29 PM
 Subject: Re: Kafka JMS metrics meaning
   
Under replicated is a must. Offline partitions is also good to monitor. We also use the active
controller metric (it's 1 or 0) in aggregate for a cluster to know that the controller is
running somewhere. 

For more general metrics, all topics bytes in and bytes out is good. We also watch the leader
partitions count to know when to do a preferred replica election. Specifically, we take the
ratio of that number to the total partition count for the broker and keep it near 50%

Most other things, like specific request type time and 99% metrics, we generally only look
at when we are doing performance testing or have a specific concern. 

-Todd



> On Jun 2, 2015, at 1:01 PM, Aditya Auradkar <aauradkar@linkedin.com.INVALID> wrote:
> 
> Number of underreplicated partitions, total request time are some good bets.
> 
> Aditya
> 
> ________________________________________
> From: Otis Gospodnetic [otis.gospodnetic@gmail.com]
> Sent: Tuesday, June 02, 2015 9:56 AM
> To: users@kafka.apache.org; Marina
> Subject: Re: Kafka JMS metrics meaning
> 
> Hi,
> 
>> On Tue, Jun 2, 2015 at 12:50 PM, Marina <ppine7@yahoo.com.invalid> wrote:
>> 
>> Hi,
>> I have enabled JMX_PORT for KAfka server and am trying to understand some
>> of the metrics that are being exposed. I have two questions:
>> 1. what are the best metrics to monitor to quickly spot unhealthy Kafka
>> cluster?
> 
> People loooove looking at consumer lag :)
> 
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
> 
> 2. what do these metrics mean: ReplicaManager -> LeaderCount ? and
>> ReplicaManager -> PartitionCount ?I have three topics created, with one
>> partition each, and replication = 1, however the values for both of the
>> above attributes is "53".... So I am not sure what the count '53' means
>> here....
>> thanksMarina
>> 

  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message