kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: Monitoring Producers at Large Scale
Date Wed, 25 Jun 2014 17:49:01 GMT
We monitor producers or for that matter any process/service using JMX
metrics. Every server and service in LinkedIn sends metrics in a Kafka
message to a metrics Kafka cluster. We have subscribers that connect to the
metrics cluster to index that data in RRDs.

Our aim is to expose all important metrics through JMX. We are doing that
for the new producer under org.apache.kafka.clients.producer. Feel free to
take a look at that and give feedback.

Thanks,
Neha


On Tue, Jun 24, 2014 at 7:59 PM, Darion Yaphet <darion.yaphet@gmail.com>
wrote:

> Sorry I want to  know  you want to monitor kafka producers or kafka brokers
> and zookeepers ?
> It's seems you will want to monitor monitor Exceptions eg Leader Not Found,
> Queue is full, resend fail  etc  are kafka cluster
>
>
> 2014-06-25 8:20 GMT+08:00 Bhavesh Mistry <mistry.p.bhavesh@gmail.com>:
>
> > We use Kafka as Transport Layer to transport application logs.  How do we
> > monitor Producers at large scales about 6000 boxes x 4 topic per box so
> > roughly 24000 producers (spread across multiple data center.. we have
> > brokers per DC).  We do the monitoring based on logs.  I have tried
> > intercepting logs via Log4J custom implementation which only intercept
> WARN
> > and ERROR and FATAL events  org.apache.log4j.AppenderSkeleton append
> method
> > which send its logs to brokers (This is working but after load testing it
> > is causing deadlock some times between ProducerSendThread and Producer).
> >
> > I know there are JMX monitoring MBeans available which we can pull the
> > data, but I would like to monitor Exceptions eg Leader Not Found, Queue
> is
> > full, resend fail etc in Kafka Library.
> >
> > How does LinkedIn monitor the Producers ?
> >
> > Thanks,
> >
> > Bhavesh
> >
>
>
>
> --
>
>
> long is the way and hard  that out of Hell leads up to light
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message