storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis DEBARBIEUX <ddebarbi...@norsys.fr>
Subject Re: Problem when consuming storm metrics
Date Tue, 26 Apr 2016 11:16:40 GMT
Dear all,

Please find enclose the full thread dump. I hope that you can find some 
promishing information.

I did the following experiments:
  - Reproduce the problem with the ExclamationTopology (storm starter). 
But I fail.
  - Activate acking (there is no acking in my toplogy). But I still have 
the problem.

Sorry for the delay between my two posts.
Thanks for our help.

Denis

Le 11/04/2016 13:47, Abhishek Agarwal a écrit :
> I wanted to look at bolt stack specifically. With the names removed, 
> it is difficult to figure out. can you tell me what are the 
> bolts/spouts busy in? Are they very slow? The timer which publishes 
> metric tick is blocked because the receive queue of some executor is 
> full.
>
> Some related details are captured here - 
> https://issues.apache.org/jira/browse/STORM-972
>
> On Mon, Apr 11, 2016 at 2:28 PM, Denis DEBARBIEUX 
> <ddebarbieux@norsys.fr <mailto:ddebarbieux@norsys.fr>> wrote:
>
>     Jungtaek,
>
>     Thanks for your help.
>
>     Please find enclose the jstack file:
>      - I remove by hand all name related to my project (spout, bolt...).
>      - The running jdk was openJDK (not an oracle JDK).
>
>     I also increase the buffer size
>     (topology.executor.receive.buffer.size and
>     topology.executor.send.buffer.size)  to 32768 and lauch my
>     topology on 2 workers. Then the dead lock occurs after 3 days. At
>     this moment 24577 points have been send.
>
>     Please feel free to contact me if you need further information.
>
>     Denis
>
>     Le 29/03/2016 11:11, Jungtaek Lim a écrit :
>>     Denis,
>>
>>     Could you share your jstack output which contains locked information?
>>     And I'm not sure but you may hit STORM-1027
>>     <https://issues.apache.org/jira/browse/STORM-1027> which is
>>     resolved at 0.10.0 and 0.9.6.
>>
>>     Thanks,
>>     Jungtaek Lim (HeartSaVioR)
>>
>>     2016년 3월 29일 (화) 오후 5:41, Denis DEBARBIEUX
>>     <ddebarbieux@norsys.fr <mailto:ddebarbieux@norsys.fr>> 님이 작성:
>>
>>         Hi all,
>>
>>         I profile my code and I see that the thread
>>         <BoldName>MetricConsumer is locked and never unlocked. It is
>>         why my metrics are never published.
>>
>>         More over, the problem is related to buffer size:
>>         * topology.executor.receive.buffer.size=16 and
>>         topology.executor.send.buffer.size=16 implies a blocking
>>         after 4 minutes.
>>         * with a size=512, I am able to run my test for more than 1 hour.
>>
>>
>>         Thanks for your help.
>>
>>         Denis
>>
>>         Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
>>>
>>>         Do you get this problem if you use only the
>>>         LoggingMetricsConsumer?
>>>
>>>         Excuse typos
>>>
>>>         On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX"
>>>         <ddebarbieux@norsys.fr <mailto:ddebarbieux@norsys.fr>> wrote:
>>>
>>>             Dear all,
>>>
>>>             I have a question about metrics and how they are consumed:
>>>             I launch my topology. During 3 or 4 minutes, I receive
>>>             my metrics. Then no more logs.
>>>             It's look like the process that is consuming the metrics
>>>             has a dead lock or is blocked by a time out.
>>>
>>>             Here comes some information about the context:
>>>              - I am using storm 0.9.4.
>>>              - Some metrics are defined by defaut (eg
>>>             emit-count.default) some other are customed.
>>>              - I am sure that my topology is still running.
>>>              - According to Storm UI, the number of messages emiting
>>>             by my bolts is increasing. In other words, metrics are
>>>             still computing and send somewhere.
>>>              - I try two instances of LoggingMetricsConsumer: in
>>>             file (backtype.storm.metric.LoggingMetricsConsumer) and
>>>             in graphite
>>>             (com.verisign.storm.metrics.GraphiteMetricsConsumer).
>>>              - I can reproduce it on 2 different machines.
>>>              - No error are logged (worker, metric, supervisor).
>>>              - I am using storm since 1 year and I never seen this
>>>             problem.
>>>
>>>             Thanks for your help.
>>>
>>>             Denis
>>>
>>>             ---
>>>             L'absence de virus dans ce courrier électronique a été
>>>             vérifiée par le logiciel antivirus Avast.
>>>             https://www.avast.com/antivirus
>>>
>>
>>
>>
>>         ------------------------------------------------------------------------
>>         Avast logo
>>         <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>         	
>>
>>         L'absence de virus dans ce courrier électronique a été
>>         vérifiée par le logiciel antivirus Avast.
>>
>>         www.avast.com <http://www.avast.com>
>>
>>
>
>
>
>     ------------------------------------------------------------------------
>     Avast logo
>     <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>     	
>
>     L'absence de virus dans ce courrier électronique a été vérifiée
>     par le logiciel antivirus Avast.
>     www.avast.com
>     <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>
>
>
>
> -- 
> Regards,
> Abhishek Agarwal
>



---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus
Avast.
https://www.avast.com/antivirus

Mime
View raw message