storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Lowe <githubpoi...@gmail.com>
Subject Re: Storm monitoring
Date Tue, 31 May 2016 13:19:04 GMT
Thought you all might be interested.

I have now got capacity monitoring working with AWS Cloudwatch. These are 2
bolts within the same topology:

‚Äč


#! /bin/bash

API_KEY='XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

# get all the topology id's
TOPOLOGY_IDS=`curl -s
http://XXXXXXXXXXXXXXXXXXXXXXXX:8080/api/v1/topology/summary | json_pp |
grep '"id"' | sed 's/.* : "\(.*\)".*/\1/'`
for id in ${TOPOLOGY_IDS} ; do
TOPOLOGY=`curl -s
http://XXXXXXXXXXXXXXXXXXXXXXX:8080/api/v1/topology/${id}?sys=false`

# check the capacity for each task
CAPACITIES=`echo ${TOPOLOGY} | json_pp | grep '"capacity"' | sed 's/.* :
"\(.*\)".*/\1/'`
CAPACITIES_ARRAY=()
for capactiy in ${CAPACITIES} ; do
CAPACITIES_ARRAY+=("${capactiy}")
done

# get all of the bolt ids
BOLT_IDS=`echo ${TOPOLOGY} | json_pp | grep '"boltId"' | sed 's/.* :
"\(.*\)".*/\1/'`
BOLT_IDS_ARRAY=()
for boltId in ${BOLT_IDS} ; do
BOLT_IDS_ARRAY+=("${boltId}")
done

    # send a trigger if needed
    # capacity and bolt id arrays will always be the same size
for(( x=0; x<${#CAPACITIES_ARRAY[@]}; x++ ));  do
TIMESTAMP=`date +"%Y-%m-%dT%H:%M:%S.%N"`
/usr/local/bin/aws cloudwatch put-metric-data --metric-name
"${BOLT_IDS_ARRAY[$x]}-capacity" --namespace "${id}" --value
"${CAPACITIES_ARRAY[$x]}" --timestamp "${TIMESTAMP}"
done


done

On Mon, May 30, 2016 at 5:37 PM, Radhwane Chebaane <r.chebaane@mindlytix.com
> wrote:

>
> Using graphite API was so helpful for us, but since it doesn't support
> *Tags* introduced since *InfluxDB 0.9*, we created a *new metric Consumer
> <https://github.com/mathieuboniface/storm-metrics-influxdb>* that
> supports the new InfluxDB API instead of Grahpite API. *Tags* are
> important when filtering dashboard metrics based on components, bolt or
> worker name.
>
> Regards,
> Radhwane
>
>
> On 30/05/2016 17:19, Stephen Powis wrote:
>
> +1 for graphite and grafana via Verisign's plugin.
>
> Using graphite a few years ago was a real game changer for us, and more
> recently grafana to help build out dashboards instead of copy/pasting
> graphite urls around.  Here's two different dashboards we have relating to
> our storm topologies.  We're able to correlate information from all parts
> of our app, hardware monitoring metrics (via zabbix) and of course storm.
> Additionally we use seyren on top of graphite for our alerting as well.
>
> bolt specific dashboard <http://i.imgur.com/ftKtci5.png>
>
> Dashboard correlating lots of related information from various sources
> <http://i.imgur.com/t7yJ8d5.jpg>
>
>
>
> On Mon, May 30, 2016 at 9:13 AM, Radhwane Chebaane <
> <r.chebaane@mindlytix.com>r.chebaane@mindlytix.com> wrote:
>
>> Hi Matthew,
>>
>> We actually use the InfluxData <https://influxdata.com/> Stack (InfluxDb
>> + Grafana).
>> We send our data directly to a time-series database, *InfluxDB*
>> <https://influxdata.com/time-series-platform/influxdb/>. Then, we
>> visualize metrics with a customizable dashboard, *Grafana*
>> <http://grafana.org/>.
>> This way, you can have real-time metrics on your Storm topology. You may
>> also add custom metrics for enhanced monitoring.
>>
>> To export Storm metrics to InfluxDB you can use this *MetricsConsumer *which
>> is compatible with the latest version of InfluxDB and Storm 1.0.0:
>> https://github.com/mathieuboniface/storm-metrics-influxdb
>>
>> Or you can use the old Verisign plug-in with Graphite protocol:
>> https://github.com/verisign/storm-graphite
>>
>> Best regards,
>> Radhwane CHEBAANE
>>
>>
>> On 30/05/2016 14:47, Matthew Lowe wrote:
>>
>> Hello all.
>>
>> What kind of monitoring solutions do you use with storm?
>>
>> For example I have a bash script that reads the Json data from the REST UI and alerts
if there are any bolts with high capacities.
>>
>> It's only small and hacky, but I am genuinely interested to how you all monitor your
topologies.
>>
>> Best Regards
>> Matthew Lowe
>>
>>
>>
>
>

Mime
View raw message