storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cody A. Ray" <cody.a....@gmail.com>
Subject Re: Position in Kafka Stream
Date Wed, 28 May 2014 14:20:15 GMT
You can also use stormkafkamon to track this stuff. Its not good for
historical analysis like graphite/ganglia, but its good if you just want to
see how things currently stand.

The original: https://github.com/otoolep/stormkafkamon

This didn't work for us without some updates (incompatibility with the
latest python-kafka dep). Here are those updates:
https://github.com/BrightTag/stormkafkamon/commit/07eede9ec72329fe2cad893d087541b583e11148

(Our branch has a couple more things that parse the kafka messages with our
format (which embeds a timestamp) to determine how long (in time) storm is
behind... planning to clean that up soon so it can be a bit more reusable)

https://github.com/BrightTag/stormkafkamon

-Cody


On Wed, May 28, 2014 at 4:50 AM, Danijel Schiavuzzi
<danijel@schiavuzzi.com>wrote:

> Yes, Trident Kafka spouts give you the same metrics. Take a look at the
> code to find out what's available.
>
>
> On Wed, May 28, 2014 at 3:55 AM, Tyson Norris <tnorris@adobe.com> wrote:
>
>> Do Trident variants of kafka spouts do something similar?
>> Thanks
>> Tyson
>>
>> > On May 27, 2014, at 3:19 PM, "Harsha" <storm@harsha.io> wrote:
>> >
>> > Raphael,
>> >        kafka spout sends metrics for kafkaOffset and kafkaPartition you
>> can look at those by using LoggingMetrics or setting up a ganglia. Kafka
>> uses its own zookeeper to store state info per topic & group.id you can
>> look at kafka offsets using
>> > kafka/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
>> > -Harsha
>> >
>> >
>> >> On Tue, May 27, 2014, at 03:01 PM, Raphael Hsieh wrote:
>> >> Is there a way to tell where in the kafka stream my topology is
>> starting from?
>> >> From my understanding Storm will use zookeeper in order to tell its
>> place in the Kafka stream. Where can I find metrics on this ?
>> >> How can I see how large the stream is? What how much data is sitting
>> in the stream and what the most recent/oldest position is?
>> >>
>> >> Thanks
>> >>
>> >> --
>> >> Raphael Hsieh
>>
>
>
>
> --
> Danijel Schiavuzzi
>
> E: danijel@schiavuzzi.com
> W: www.schiavuzzi.com
> T: +385989035562
> Skype: danijels7
>



-- 
Cody A. Ray, LEED AP
cody.a.ray@gmail.com
215.501.7891

Mime
View raw message