storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis DEBARBIEUX <ddebarbi...@norsys.fr>
Subject Re: Problem when consuming storm metrics
Date Mon, 11 Apr 2016 08:58:56 GMT
Jungtaek,

Thanks for your help.

Please find enclose the jstack file:
  - I remove by hand all name related to my project (spout, bolt...).
  - The running jdk was openJDK (not an oracle JDK).

I also increase the buffer size (topology.executor.receive.buffer.size 
and topology.executor.send.buffer.size)  to 32768 and lauch my topology 
on 2 workers. Then the dead lock occurs after 3 days. At this moment 
24577 points have been send.

Please feel free to contact me if you need further information.

Denis

Le 29/03/2016 11:11, Jungtaek Lim a écrit :
> Denis,
>
> Could you share your jstack output which contains locked information?
> And I'm not sure but you may hit STORM-1027 
> <https://issues.apache.org/jira/browse/STORM-1027> which is resolved 
> at 0.10.0 and 0.9.6.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 3월 29일 (화) 오후 5:41, Denis DEBARBIEUX 
> <ddebarbieux@norsys.fr <mailto:ddebarbieux@norsys.fr>> 님이 작성:
>
>     Hi all,
>
>     I profile my code and I see that the thread
>     <BoldName>MetricConsumer is locked and never unlocked. It is why
>     my metrics are never published.
>
>     More over, the problem is related to buffer size:
>     * topology.executor.receive.buffer.size=16 and
>     topology.executor.send.buffer.size=16 implies a blocking after 4
>     minutes.
>     * with a size=512, I am able to run my test for more than 1 hour.
>
>
>     Thanks for your help.
>
>     Denis
>
>     Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
>>
>>     Do you get this problem if you use only the LoggingMetricsConsumer?
>>
>>     Excuse typos
>>
>>     On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX"
>>     <ddebarbieux@norsys.fr <mailto:ddebarbieux@norsys.fr>> wrote:
>>
>>         Dear all,
>>
>>         I have a question about metrics and how they are consumed:
>>         I launch my topology. During 3 or 4 minutes, I receive my
>>         metrics. Then no more logs.
>>         It's look like the process that is consuming the metrics has
>>         a dead lock or is blocked by a time out.
>>
>>         Here comes some information about the context:
>>          - I am using storm 0.9.4.
>>          - Some metrics are defined by defaut (eg emit-count.default)
>>         some other are customed.
>>          - I am sure that my topology is still running.
>>          - According to Storm UI, the number of messages emiting by
>>         my bolts is increasing. In other words, metrics are still
>>         computing and send somewhere.
>>          - I try two instances of LoggingMetricsConsumer: in file
>>         (backtype.storm.metric.LoggingMetricsConsumer) and in
>>         graphite (com.verisign.storm.metrics.GraphiteMetricsConsumer).
>>          - I can reproduce it on 2 different machines.
>>          - No error are logged (worker, metric, supervisor).
>>          - I am using storm since 1 year and I never seen this problem.
>>
>>         Thanks for your help.
>>
>>         Denis
>>
>>         ---
>>         L'absence de virus dans ce courrier électronique a été
>>         vérifiée par le logiciel antivirus Avast.
>>         https://www.avast.com/antivirus
>>
>
>
>
>     ------------------------------------------------------------------------
>     Avast logo
>     <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>     	
>
>     L'absence de virus dans ce courrier électronique a été vérifiée
>     par le logiciel antivirus Avast.
>
>     www.avast.com
>     <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>



---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus
Avast.
https://www.avast.com/antivirus

Mime
View raw message