kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arunkumar <pm_arunku...@yahoo.com.INVALID>
Subject Re: Monitoring and Alerting
Date Fri, 27 Apr 2018 02:07:45 GMT
Hi All
Thank you for your time and effort. All have suggested my to look at Prometheus etc... I
am not looking for a solution. I need metrics on which I can do alerting. We are implementing
custom alerting mechanism. I am using JMX to get all the metrics but as many of you suggested
I do not need all of them. I need very few to alert and I need details about that metrics
on which I can create alerts.
Thanks in advance 
ThanksArunkumar Pichaimuthu, PMP 

    On Wednesday, April 25, 2018, 11:35:27 PM CDT, Anand, Uttam <Uttam.Anand@BNSF.com>
wrote:  
 
 You can use Prometheus for monitoring , AlertManager for alerting and Grafana to view the
real time graphs by using the metrics scraped by Prometheus.


For gathering the metrics of Kafka and Zookeeper , you can use jmx exporter. Also, all metrics
will not be useful for humans to read and understand and it will be overhead for monitoring
apps to process them. So you can whitelist the metrics you actually want.


https://www.robustperception.io/monitoring-kafka-with-prometheus


You can add jmx exporter as environment variable in container also.
<https://www.robustperception.io/monitoring-kafka-with-prometheus>

Monitoring Kafka with Prometheus | Robust Perception<https://www.robustperception.io/monitoring-kafka-with-prometheus/>
www.robustperception.io
We’ve previously looked at how to monitor Cassandra with Prometheus. Let’s see the process
for getting metrics from another popular Java application, Kafka. Similar to what we did for
Cassandra, we download Kafka, the JMX exporter and the config file:




________________________________
From: Will Weber <rwaweber@gmail.com>
Sent: Wednesday, April 25, 2018 11:27:40 PM
To: users@kafka.apache.org
Subject: Re: Monitoring and Alerting

<center><p style="color:red; font-size:24px;"><b>EXTERNAL EMAIL</b></p></center>
"Thirding" the use of prometheus, recommend reading this blog post as well:
https://www.robustperception.io/monitoring-kafka-with-prometheus

On Wed, Apr 25, 2018 at 11:39 PM, Jonathan Bethune <
jonathan.bethune@instaclustr.com> wrote:

> I also recommend Prometheus. Works great with JMX for Kafka or any Java
> service.
>
> Datadog is also fine if you want simple and don't spending money. Easy to
> integrate with alerting systems and great visualization.
>
> They have a good blogpost about integrating with Kafka and ZooKeeper:
> https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/
>
> On 26 April 2018 at 08:02, Stanislav Antic <stanislav.antic@gmail.com>
> wrote:
>
> > I recommend setting up Prometheus which have Zookeeper exporter (helper
> > program which gives metrics from outside software) and you can use JMX
> > exporter with Kafka.
> >
> > You can find example configs in their repo, which is pretty good and you
> > also have already done Grafana dashboards (https://grafana.com/
> dashboards
> > ):
> > https://github.com/prometheus/jmx_exporter/tree/master/example_configs
> >
> >
> > On Thu, Apr 26, 2018 at 12:05 AM, Arunkumar <pm_arunkumar@yahoo.com.
> > invalid>
> > wrote:
> >
> > > HI All
> > > I am working on setting up Monitoring and alerting for our production
> > > cluster. As of now we have a cluster of 3 zookeeper and 3 kafka Brokers
> > > which will expand later.
> > > We are planning for basic metrics (important ones) on which we need to
> > > alert. We are in a process of developing alerting system for our
> > cluster. I
> > > googled and did not get much detail as needed.
> > > I see many say the following ones are important and I understand these
> > are
> > > aggregate metrics across cluster (I may not be
> right).kafka.server:type=
> > > ReplicaManager,name=UnderReplicatedPartitionskafka.controller:type=
> > > KafkaController,name=OfflinePartitionsCountkafka.controller:type=
> > > KafkaController,name=ActiveControllerCount
> > > But I am looking for broker,topic etc metrics to alert. Can any one
> > > provide info or point to any docs on this regards will be of great
> > > help.            Thanks in advance
> > >
> > > ThanksArunkumar Pichaimuthu, PMP
> >
> >
> >
> >
> > --
> > Stanislav Antic
> >
>
>
>
> --
>
> *Jonathan Bethune - **Senior Consultant*
>
> JP: +81 70 4069 4357
>
> <https://www.facebook.com/instaclustr>  <https://twitter.com/instaclustr>
> <https://www.linkedin.com/company/instaclustr>
>
> Read our latest technical blog posts here
> <https://www.instaclustr.com/blog/>. This email has been sent on behalf
> of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This
> email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message