storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsha <>
Subject Re: [DISCUSS] Would like to make collective intelligence about Metrics on Storm
Date Tue, 03 May 2016 05:47:35 GMT
I think filters that can support a regex gives more felxibility.
On Mon, May 2, 2016, at 07:48 PM, Jungtaek Lim wrote:
> Kevin,
> For specific task, you can register your own metrics which resides
> per task.
> But metrics doc on Storm is not kind enough to let users follow, so I
> addressed this and submitted pull request.
> There're no custom worker metrics for users since Storm abstracts
> user's logic into 'task' so normally users don't need to measure JVM
> level metrics (except JMX metrics). But it would be possible to add if
> it's really great.
> Does current Storm work for you now? Or we would like to address /
> improve something?
> Stephen and Harsha,
> Improve MetricsConsumer is an umbrella issue, and it contains the
> feature 'blacklist & whitelist of metrics[1]'.
> For now it filters metrics by only metrics name, but adding filter
> targets to component name, host, etc. are easy so I would like to see
> the needs before making the change.
> Do you think adding filter targets makes sense? Or we can just start
> with metrics name filter?
> Thanks,
> Jungtaek Lim (HeartSaVioR)
> 2016년 5월 3일 (화) 오전 2:18, Harsha <>님이 작성:
>> __
>> Jungtaek,
>> Probably a filter config to whitelist and blacklist certain metrics.
>> So that it will scale if there are too many workers and users can
>> turn off certain metrics.
>> Thanks,
>> Harsha
>> On Mon, May 2, 2016, at 06:19 AM, Stephen Powis wrote:
>>> Oooh I'd love this as well!  I really dig the ease of the metric
>>> framework in storm and have all the metrics go thru one centralized
>>> config.  But as the number of storm hosts and number of tasks grow,
>>> I've found that Graphite/Grafana has a hard time collecting up all
>>> the relevant metrics across a lot of wildcarded keys for things like
>>> hostnames and taskIds to properly display my graphs.
>>> On Sun, May 1, 2016 at 8:17 AM, Kevin Conaway
>>> <> wrote:
>>>> One thing I would like to see added (if not already present) is the
>>>> ability to register metrics that are not tied to a component.
>>>> As of now, the only non-component metrics are reported by the
>>>> SystemBolt pseudo-component which feels like a work-around.  It
>>>> reports JVM level metrics like GC time, heap size and other things
>>>> that aren't associated with a given component.
>>>> It would be great if application developers could expose similar
>>>> metrics like this for things like connection pools and other JVM
>>>> wide objects that aren't unique to a specific component.
>>>> I don't think this is possible now, is it?
>>>> On Wed, Apr 20, 2016 at 12:29 AM, Jungtaek Lim <>
>>>> wrote:
>>>>> Let me start sharing my thought. :)
>>>>> 1. Need to enrich docs about metrics / stats.
>>>>> In fact, I couldn't see the fact - topology stats are sampled by
>>>>> default and sample rate is 0.05 - from the docs when I was newbie
>>>>> of Apache Storm. It made me misleading and made me saying "Why
>>>>> there're difference between the counts?". I also saw some mails
>>>>> from user@ about same question. If we include this to guide doc
>>>>> that would be better.
>>>>> And Metrics document page[2] seems not well written. I think it
>>>>> has appropriate headings but lacks contents on each heading.
>>>>> It should be addressed, and introducing some external metrics
>>>>> consumer plugins (like storm-graphite[3] from Verisign) would be
>>>>> great, too.
>>>>> 2. Need to increase sample rate or (ideally) no sampling at all.
>>>>> Let's postpone considering performance hit at this time.
>>>>> Ideally, we expect precision of metrics gets better when we
>>>>> increase sample rate. It affects non-gauge kinds of metrics which
>>>>> are counter, and latency, and so on.
>>>>> Btw, I would like to hear about opinions on latency since I'm not
>>>>> an expert.
>>>>> Storm provides only average latency and it's indeed based on
>>>>> sample rate. Do we feel OK with this? If not how much having also
>>>>> percentiles can help us?
>>>>> Thanks,
>>>>> Jungtaek Lim (HeartSaVioR)
>>>>> 2016년 4월 20일 (수) 오전 10:55, Jungtaek Lim <>님이
>>>>>> Hi Storm users,
>>>>>> I'm Jungtaek Lim, committer and PMC member of Apache Storm.
>>>>>> If you subscribed dev@ mailing list, you may have seen that
>>>>>> recently we're addressing the metrics feature on Apache Storm.
>>>>>> For now, improvements are going forward based on current metrics
>>>>>> feature.
>>>>>> - Improve (Topology) MetricsConsumer[4]
>>>>>> - Provide topology metrics in detail (metrics per each stream)[5]
>>>>>> - (WIP) Introduce Cluster Metrics Consumer
>>>>>> As I don't maintain large cluster for myself, I really want to
>>>>>> collect the any ideas for improving, any inconveniences, use
>>>>>> cases of Metrics with community members, so we're on the right
>>>>>> way to go forward.
>>>>>> Let's talk!
>>>>>> Thanks in advance,
>>>>>> Jungtaek Lim (HeartSaVioR)
>>>> --
>>>> Kevin Conaway



View raw message