kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis.gospodne...@gmail.com>
Subject Re: Anouncing Kafka Offset Monitor 0.1
Date Mon, 10 Mar 2014 20:04:50 GMT
Interesting.  We were able to get this from JMX for Kafka 0.7.2 - here's a
snapshot for one of our Kafka clusters:
https://apps.sematext.com/spm-reports/s/93EtvhnOz0

Is getting this from ZK instead of JMX better?

Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/


On Sat, Mar 8, 2014 at 7:33 AM, Pierre Andrews <pierre@quantifind.com>wrote:

> It's been developed with 0.8 in mind. Giving a quick look at how 0.7 stores
> data in ZK, it looks like it's slightly different and might not work
> straight away.
>
> If you are intersted, have a look at:
>
> https://github.com/quantifind/KafkaOffsetMonitor/blob/master/src/main/scala/com/quantifind/kafka/OffsetGetter.scala
>
> This is where the data is read from ZK and Kafka. If you have a 0.7 setup,
> you might be able to adapt it to get the right offsets, you would probably
> not have to change anything else in the code.
>
> P
>
>
> On Sat, Mar 8, 2014 at 2:48 AM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com
> > wrote:
>
> > Hi,
> >
> > Does this work with Kafka 0.7.x or does Kafka 0.7.x not expose the info
> > needed for computing the lag?
> >
> > Thanks,
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Fri, Mar 7, 2014 at 1:49 PM, Pierre Andrews <pierre@quantifind.com
> > >wrote:
> >
> > > Hello everyone,
> > >
> > > at Quantifind, we are big users of Kafka and we like it a lot!
> > > In a few use cases, we had to figure out if a queue was growing and how
> > its
> > > consumers were behaving. There are a few command-line tools to try to
> > > figure out what's going on, but it's not always easy to debug and to
> see
> > > what has happened while everyone was sleeping.
> > >
> > > To be able to monitor our kafka queues and consumers, we thus
> developed a
> > > tiny web app that could tell us the log size of each topic in the
> brokers
> > > and the offsets of each consumers. That's very similar to what the
> kafka
> > > ConsumerOffsetChecker tool* is doing, but instead of having a one off
> > > snapshot, our app keeps an history and displays a nice graph of what's
> > > going on.
> > >
> > > You can find screenshots and more details here:
> > > http://quantifind.github.io/KafkaOffsetMonitor/
> > >
> > > the code is on github:
> > > https://github.com/quantifind/KafkaOffsetMonitor
> > >
> > > If you have kafka 0.8 setup, it's very easy to use:
> > >
> > >   1- download the current jar
> > >
> > >
> >
> http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
> > >   2- run it, pointing at your kafka brokers:
> > >     java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
> > >      com.quantifind.kafka.offsetapp.OffsetGetterWeb \
> > >      --zk zk-server1,zk-server2 \
> > >      --port 8080 \
> > >      --refresh 10.seconds \
> > >      --retain 2.days
> > >    3- open your browser and point it to localhost:8080
> > >
> > > You can run it locally or host it on a server if you prefer.
> > >
> > > It's all open source and we'll be happy to receive issue report and
> pull
> > > requests.
> > >
> > > I hope that you like it and that it can find some uses in the rest of
> the
> > > Kafka community.
> > >
> > > Best
> > >
> > > Pierre
> > >
> > > PS: we are aware of https://github.com/claudemamo/kafka-web-consolebut
> > > are
> > > currently offering slightly different features. Hopefully we can merge
> > the
> > > projects in the future.
> > >
> > > * here:
> > >
> > >
> >
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message