kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre Andrews <pie...@quantifind.com>
Subject Re: Anouncing Kafka Offset Monitor 0.1
Date Sat, 08 Mar 2014 12:29:29 GMT
Hi Patricio,

thank you!

I don't know how kafka-spout works. Right now, the monitor uses the value
defined in kafka.utils.ZkUtils.ConsumersPath to grab the list of consumers
in ZK. Again, that could be a parameter to the app somewhere to be more
flexible and deal with separate zk path.

P


On Sat, Mar 8, 2014 at 12:42 AM, Patricio Echagüe <patricioe@gmail.com>
wrote:
>
> Great work!
>
> In addition to Dan's question, does it work with storm kafka-spout which
> uses a separate zk path?
>
> Sent from my Nexus 4.
> On Mar 7, 2014 7:18 PM, "Dan Hoffman" <hoffmandan@gmail.com> wrote:
>
> > Hi - am I right in that for this tool to be effective, consumers must be
> > using the high level consumer or otherwise keeping their offsets in
> > zookeeper?  Is there any way to track performance without that?
> >
> >
> > On Fri, Mar 7, 2014 at 3:08 PM, Steve Morin <steve.morin@gmail.com>
wrote:
> >
> > > Very nice
> > >
> > > > On Mar 7, 2014, at 11:55, Pierre Andrews <pierre@quantifind.com>
> > wrote:
> > > >
> > > > Claude, we should join forces ;)
> > > >
> > > >
> > > >> On Fri, Mar 7, 2014 at 4:45 PM, Claude Mamo <claude.mamo@gmail.com>
> > > wrote:
> > > >>
> > > >> Awesome!!! ;-)
> > > >>
> > > >> Claude
> > > >>
> > > >>
> > > >> On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews <
pierre@quantifind.com
> > > >>> wrote:
> > > >>
> > > >>> Great! Thanks!
> > > >>>
> > > >>>
> > > >>>> On Fri, Mar 7, 2014 at 3:59 PM, Jay Kreps <jay.kreps@gmail.com>
> > > wrote:
> > > >>>>
> > > >>>> This is really useful! I added it to the ecosystem page:
> > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
> > > >>>>
> > > >>>> -Jay
> > > >>>>
> > > >>>>
> > > >>>> On Fri, Mar 7, 2014 at 10:49 AM, Pierre Andrews <
> > > pierre@quantifind.com
> > > >>>>> wrote:
> > > >>>>
> > > >>>>> Hello everyone,
> > > >>>>>
> > > >>>>> at Quantifind, we are big users of Kafka and we like it
a lot!
> > > >>>>> In a few use cases, we had to figure out if a queue was
growing
and
> > > >> how
> > > >>>> its
> > > >>>>> consumers were behaving. There are a few command-line
tools to
try
> > to
> > > >>>>> figure out what's going on, but it's not always easy to
debug
and
> > to
> > > >>> see
> > > >>>>> what has happened while everyone was sleeping.
> > > >>>>>
> > > >>>>> To be able to monitor our kafka queues and consumers,
we thus
> > > >>> developed a
> > > >>>>> tiny web app that could tell us the log size of each topic
in
the
> > > >>> brokers
> > > >>>>> and the offsets of each consumers. That's very similar
to what
the
> > > >>> kafka
> > > >>>>> ConsumerOffsetChecker tool* is doing, but instead of having
a
one
> > off
> > > >>>>> snapshot, our app keeps an history and displays a nice
graph of
> > > >> what's
> > > >>>>> going on.
> > > >>>>>
> > > >>>>> You can find screenshots and more details here:
> > > >>>>> http://quantifind.github.io/KafkaOffsetMonitor/
> > > >>>>>
> > > >>>>> the code is on github:
> > > >>>>> https://github.com/quantifind/KafkaOffsetMonitor
> > > >>>>>
> > > >>>>> If you have kafka 0.8 setup, it's very easy to use:
> > > >>>>>
> > > >>>>>  1- download the current jar
> > > >>
> > >
> >
http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
> > > >>>>>  2- run it, pointing at your kafka brokers:
> > > >>>>>    java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
\
> > > >>>>>     com.quantifind.kafka.offsetapp.OffsetGetterWeb \
> > > >>>>>     --zk zk-server1,zk-server2 \
> > > >>>>>     --port 8080 \
> > > >>>>>     --refresh 10.seconds \
> > > >>>>>     --retain 2.days
> > > >>>>>   3- open your browser and point it to localhost:8080
> > > >>>>>
> > > >>>>> You can run it locally or host it on a server if you prefer.
> > > >>>>>
> > > >>>>> It's all open source and we'll be happy to receive issue
report
and
> > > >>> pull
> > > >>>>> requests.
> > > >>>>>
> > > >>>>> I hope that you like it and that it can find some uses
in the
rest
> > of
> > > >>> the
> > > >>>>> Kafka community.
> > > >>>>>
> > > >>>>> Best
> > > >>>>>
> > > >>>>> Pierre
> > > >>>>>
> > > >>>>> PS: we are aware of
> > > >> https://github.com/claudemamo/kafka-web-consolebut
> > > >>>>> are
> > > >>>>> currently offering slightly different features. Hopefully
we can
> > > >> merge
> > > >>>> the
> > > >>>>> projects in the future.
> > > >>>>>
> > > >>>>> * here:
> > > >>
> > >
> >
https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
> > > >>
> > >
> >

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message