storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhishek Agarwal <abhishc...@gmail.com>
Subject Re: Problem when consuming storm metrics
Date Mon, 11 Apr 2016 11:47:14 GMT
I wanted to look at bolt stack specifically. With the names removed, it is
difficult to figure out. can you tell me what are the bolts/spouts busy in?
Are they very slow? The timer which publishes metric tick is blocked
because the receive queue of some executor is full.

Some related details are captured here -
https://issues.apache.org/jira/browse/STORM-972

On Mon, Apr 11, 2016 at 2:28 PM, Denis DEBARBIEUX <ddebarbieux@norsys.fr>
wrote:

> Jungtaek,
>
> Thanks for your help.
>
> Please find enclose the jstack file:
>  - I remove by hand all name related to my project (spout, bolt...).
>  - The running jdk was openJDK (not an oracle JDK).
>
> I also increase the buffer size (topology.executor.receive.buffer.size and
> topology.executor.send.buffer.size)  to 32768 and lauch my topology on 2
> workers. Then the dead lock occurs after 3 days. At this moment 24577
> points have been send.
>
> Please feel free to contact me if you need further information.
>
> Denis
>
> Le 29/03/2016 11:11, Jungtaek Lim a écrit :
>
> Denis,
>
> Could you share your jstack output which contains locked information?
> And I'm not sure but you may hit STORM-1027
> <https://issues.apache.org/jira/browse/STORM-1027> which is resolved at
> 0.10.0 and 0.9.6.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 3월 29일 (화) 오후 5:41, Denis DEBARBIEUX < <ddebarbieux@norsys.fr>
> ddebarbieux@norsys.fr> 님이 작성:
>
>> Hi all,
>>
>> I profile my code and I see that the thread <BoldName>MetricConsumer is
>> locked and never unlocked. It is why my metrics are never published.
>>
>> More over, the problem is related to buffer size:
>> * topology.executor.receive.buffer.size=16 and
>> topology.executor.send.buffer.size=16 implies a blocking after 4 minutes.
>> * with a size=512, I am able to run my test for more than 1 hour.
>>
>>
>> Thanks for your help.
>>
>> Denis
>>
>> Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
>>
>> Do you get this problem if you use only the LoggingMetricsConsumer?
>>
>> Excuse typos
>> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX" <ddebarbieux@norsys.fr>
>> wrote:
>>
>>> Dear all,
>>>
>>> I have a question about metrics and how they are consumed:
>>> I launch my topology. During 3 or 4 minutes, I receive my metrics. Then
>>> no more logs.
>>> It's look like the process that is consuming the metrics has a dead lock
>>> or is blocked by a time out.
>>>
>>> Here comes some information about the context:
>>>  - I am using storm 0.9.4.
>>>  - Some metrics are defined by defaut (eg emit-count.default) some other
>>> are customed.
>>>  - I am sure that my topology is still running.
>>>  - According to Storm UI, the number of messages emiting by my bolts is
>>> increasing. In other words, metrics are still computing and send somewhere.
>>>  - I try two instances of LoggingMetricsConsumer: in file
>>> (backtype.storm.metric.LoggingMetricsConsumer) and in graphite
>>> (com.verisign.storm.metrics.GraphiteMetricsConsumer).
>>>  - I can reproduce it on 2 different machines.
>>>  - No error are logged (worker, metric, supervisor).
>>>  - I am using storm since 1 year and I never seen this problem.
>>>
>>> Thanks for your help.
>>>
>>> Denis
>>>
>>> ---
>>> L'absence de virus dans ce courrier électronique a été vérifiée par le
>>> logiciel antivirus Avast.
>>> https://www.avast.com/antivirus
>>>
>>>
>>
>>
>> ------------------------------
>> [image: Avast logo]
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>
>> L'absence de virus dans ce courrier électronique a été vérifiée par le
>> logiciel antivirus Avast.
>>
>>
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>> www.avast.com
>>
>>
>
>
> ------------------------------
> [image: Avast logo]
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>


-- 
Regards,
Abhishek Agarwal

Mime
View raw message