kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Hoffman <hoffman...@gmail.com>
Subject Re: Anouncing Kafka Offset Monitor 0.1
Date Sat, 08 Mar 2014 03:17:18 GMT
Hi - am I right in that for this tool to be effective, consumers must be
using the high level consumer or otherwise keeping their offsets in
zookeeper?  Is there any way to track performance without that?


On Fri, Mar 7, 2014 at 3:08 PM, Steve Morin <steve.morin@gmail.com> wrote:

> Very nice
>
> > On Mar 7, 2014, at 11:55, Pierre Andrews <pierre@quantifind.com> wrote:
> >
> > Claude, we should join forces ;)
> >
> >
> >> On Fri, Mar 7, 2014 at 4:45 PM, Claude Mamo <claude.mamo@gmail.com>
> wrote:
> >>
> >> Awesome!!! ;-)
> >>
> >> Claude
> >>
> >>
> >> On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews <pierre@quantifind.com
> >>> wrote:
> >>
> >>> Great! Thanks!
> >>>
> >>>
> >>>> On Fri, Mar 7, 2014 at 3:59 PM, Jay Kreps <jay.kreps@gmail.com>
> wrote:
> >>>>
> >>>> This is really useful! I added it to the ecosystem page:
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
> >>>>
> >>>> -Jay
> >>>>
> >>>>
> >>>> On Fri, Mar 7, 2014 at 10:49 AM, Pierre Andrews <
> pierre@quantifind.com
> >>>>> wrote:
> >>>>
> >>>>> Hello everyone,
> >>>>>
> >>>>> at Quantifind, we are big users of Kafka and we like it a lot!
> >>>>> In a few use cases, we had to figure out if a queue was growing
and
> >> how
> >>>> its
> >>>>> consumers were behaving. There are a few command-line tools to try
to
> >>>>> figure out what's going on, but it's not always easy to debug and
to
> >>> see
> >>>>> what has happened while everyone was sleeping.
> >>>>>
> >>>>> To be able to monitor our kafka queues and consumers, we thus
> >>> developed a
> >>>>> tiny web app that could tell us the log size of each topic in the
> >>> brokers
> >>>>> and the offsets of each consumers. That's very similar to what the
> >>> kafka
> >>>>> ConsumerOffsetChecker tool* is doing, but instead of having a one
off
> >>>>> snapshot, our app keeps an history and displays a nice graph of
> >> what's
> >>>>> going on.
> >>>>>
> >>>>> You can find screenshots and more details here:
> >>>>> http://quantifind.github.io/KafkaOffsetMonitor/
> >>>>>
> >>>>> the code is on github:
> >>>>> https://github.com/quantifind/KafkaOffsetMonitor
> >>>>>
> >>>>> If you have kafka 0.8 setup, it's very easy to use:
> >>>>>
> >>>>>  1- download the current jar
> >>
> http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
> >>>>>  2- run it, pointing at your kafka brokers:
> >>>>>    java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
> >>>>>     com.quantifind.kafka.offsetapp.OffsetGetterWeb \
> >>>>>     --zk zk-server1,zk-server2 \
> >>>>>     --port 8080 \
> >>>>>     --refresh 10.seconds \
> >>>>>     --retain 2.days
> >>>>>   3- open your browser and point it to localhost:8080
> >>>>>
> >>>>> You can run it locally or host it on a server if you prefer.
> >>>>>
> >>>>> It's all open source and we'll be happy to receive issue report
and
> >>> pull
> >>>>> requests.
> >>>>>
> >>>>> I hope that you like it and that it can find some uses in the rest
of
> >>> the
> >>>>> Kafka community.
> >>>>>
> >>>>> Best
> >>>>>
> >>>>> Pierre
> >>>>>
> >>>>> PS: we are aware of
> >> https://github.com/claudemamo/kafka-web-consolebut
> >>>>> are
> >>>>> currently offering slightly different features. Hopefully we can
> >> merge
> >>>> the
> >>>>> projects in the future.
> >>>>>
> >>>>> * here:
> >>
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message