storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhishek Agarwal <abhishc...@gmail.com>
Subject Re: LoggingMetricsConsumer stop logging
Date Wed, 07 Oct 2015 05:16:04 GMT
Can you check on UI that the bolt is sending tuples on the metric stream?
You have to click on "show system stats" on UI.
On Oct 6, 2015 2:34 PM, "Stepan Urban" <stepann.urban@gmail.com> wrote:

> Hi,
> I tried to increase parallelism of bolts and create some performance
> optimizations with no effect. Metrics stop logging after few hours. Is
> there some mechanism that Storm stops logging when bolts are overloaded?
> Any other sugesstions?
>
> Thanks
> Stepan
>
> 2015-10-01 19:39 GMT+02:00 Stepan Urban <stepann.urban@gmail.com>:
>
>> 1. All metrics are not being logged
>> 2. Yes topology is still prgressing, everything other works as normal,
>> worker logs are still running
>> 3. Bolt is slower, it writes data to disk, do some simple preprocessing
>> and send them to another bolt, spout only receive it and send it to bolt.
>>
>> Thanks
>> Stepan
>>
>> 2015-10-01 13:19 GMT+02:00 Abhishek Agarwal <abhishcool@gmail.com>:
>>
>>> Please check the following -
>>> 1. Are all the metrics not being logged or is it just the custom
>>> metrics? Are the in-built system metrics being logged?
>>> 2. When the metrics are not getting logged, is the topology still
>>> progressing?
>>> 3. Is your bolt too slow compared to the spout?
>>>
>>> On Thu, Oct 1, 2015 at 3:14 PM, Stepan Urban <stepann.urban@gmail.com>
>>> wrote:
>>>
>>>> Hi Ahishek,
>>>> thanks for the response. Registring metrics is normal:
>>>> transient MultiCountMetric countMetric;
>>>> transient MultiReducedMetric meanMetric;
>>>>
>>>> countMetric = new MultiCountMetric();
>>>> meanMetric = new MultiReducedMetric(new MeanReducer());
>>>> context.registerMetric("count", countMetric, 1);
>>>> context.registerMetric("mean", meanMetric, 1);
>>>>
>>>> and
>>>> Config conf = new Config();
>>>> conf.registerMetricsConsumer(LoggingMetricsConsumer.class, 1);
>>>>
>>>> even with default cluster.xml it stops logging on production computer.
>>>>
>>>> Yesterday I found that everything work on dev computer with data loaded
>>>> from file. But on production computer, when data are receided from net it
>>>> stops logging after few hours. Is it possible to find if some thread is not
>>>> working properly using jstack? Or any other way how to debug it?
>>>>
>>>> Thanks
>>>> Stepan
>>>>
>>>>
>>>>
>>>> 2015-10-01 6:26 GMT+02:00 Anishek Agarwal <anishek@gmail.com>:
>>>>
>>>>> From what i have found dont remove/add appenders in cluster.xml. not
>>>>> sure about why it works for a few hours and then stops. any specific
way
>>>>> you are registering log metrics, is it possible to put some code out
?
>>>>>
>>>>> On Tue, Sep 29, 2015 at 9:14 PM, Stepan Urban <stepann.urban@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Hi,
>>>>>> I am using LoggingMetricsConsumer and MultiReducedMetric to count
>>>>>> several metrics in Storm topology. After few hours of running it
stops
>>>>>> logging these metrics into metrics.log file without an error in any
log
>>>>>> file. Any sugestions how to find where is the problem? First I used
default
>>>>>> cluster.xml with increased size of files, now I use only
>>>>>> ch.qos.logback.core.FileAppender with the same results. After few
mb of
>>>>>> logs it stops logging.
>>>>>>
>>>>>> Thanks
>>>>>> Stepan
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Abhishek Agarwal
>>>
>>>
>>
>

Mime
View raw message