kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ravi singh <rrs120...@gmail.com>
Subject Re: Intercept broker operation in Kafka
Date Tue, 24 Jun 2014 06:51:23 GMT
Primarily we want to log below date(although this is not the exhaustive
list):

+ any error/exception during kafka start/stop
+ any error/exception while broker is running
+ broker state changes like leader re-election, broker goes down,
+ Current live brokers
+ new topic creation
+ when messages are deleted by broker after specified limit
+ Broker health : memory usage

Regards,
Ravi


On Tue, Jun 24, 2014 at 11:11 AM, Neha Narkhede <neha.narkhede@gmail.com>
wrote:

> What kind of broker metrics are you trying to push to this centralized
> logging framework?
>
> Thanks,
> Neha
> On Jun 23, 2014 8:51 PM, "ravi singh" <rrs120486@gmail.com> wrote:
>
> > Thanks Guozhang/Neha for replies.
> > Here's my use case:
> >
> > We use proprietary application logging  in our apps. We are planning to
> use
> > Kafka brokers in production , but apart from the logs which are already
> > logged using log4j in kafka we want to log the broker stats using our
> > centralized application logging framework.
> >
> > Simply put I want to write an application which could start when the
> kafka
> > brokers starts, read the broker state and metrics and push it to the
> > centralized logging servers.
> >
> > In ActiveMQ we have a plugin for our proprietary logging. We intercept
> > broker operation and install the plugin into the interceptor chain of the
> > broker.
> >
> > Regards,
> > Ravi
> >
> >
> > On Mon, Jun 23, 2014 at 9:29 PM, Neha Narkhede <neha.narkhede@gmail.com>
> > wrote:
> >
> > > Ravi,
> > >
> > > Our goal is to provide the best implementation of a set of useful
> > > abstractions and features in Kafka. The motivation behind this
> philosophy
> > > is performance and simplicity at the cost of flexibility. In most
> cases,
> > we
> > > can argue that the loss in flexibility is minimal since you can always
> > get
> > > that functionality by modeling your application differently, especially
> > if
> > > the system supports high performance. ActiveMQ has to support the JMS
> > > protocol and hence provide all sorts of hooks and plugins on the
> brokers
> > at
> > > the cost of performance.
> > >
> > > Could you elaborate more on your use case? There is probably another
> way
> > to
> > > model your application using Kafka.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Sat, Jun 21, 2014 at 9:24 AM, ravi singh <rrs120486@gmail.com>
> wrote:
> > >
> > > > How do I intercept Kakfa broker operation so that features such as
> > > > security,logging,etc can be implemented as a pluggable filter. For
> > > example
> > > > we have "BrokerFilter" class in ActiveMQ , Is there anything similar
> in
> > > > Kafka?
> > > >
> > > > --
> > > > *Regards,*
> > > > *Ravi*
> > > >
> > >
> >
> >
> >
> > --
> > *Regards,*
> > *Ravi*
> >
>



-- 
*Regards,*
*Ravi*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message