storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Taylor <>
Subject Re: 回复: RE: How to Improve Storm Application's Throughput
Date Wed, 09 Aug 2017 12:42:34 GMT

⁣Sent from BlueMail ​

On Aug 9, 2017, 8:40 AM, at 8:40 AM, "Hannum, Daniel" <>
>I think the problem is that capacity of 3.5. That indicates that
>there’s a backlog on that bolt, so it’s saying that actual time spent
>processing in the bolt is small, but the total time spent (including
>wait time) is large. Scale the bolts up or scale the spout down or make
>the bolt faster
>From: "" <>
>Reply-To: "" <>
>Date: Tuesday, August 8, 2017 at 11:02 PM
>To: user <>, libo19 <>, kabhwan
>Subject: 回复: RE: How to Improve Storm Application's Throughput
>****This email did not originate from the Premier, Inc. network. Use
>caution when opening attachments or clicking on URLs.*****
>Hi LiBo, Jungtaek :
>Yes, storm tunning depends on situations. Thank u for your kindly
>The follow is one of my situations, any hints from you will be
>appreciated. The storm version is 1.0.0.
>The topology just has a spout which reads message from kafka and a bolt
>to parse the message and put it into the hbase.
>As you can see from the above picture, the Execute latency of the bolt
>is small(0.5ms), but the Complete latency is much more larger(4365ms),
>so as to slow down the throughput of the topology.
>Which part will consume so much additional time? the transfer between
>the spout and the bolt ? or the ack part? I tried to increase
>parallelism for the component, but it did not work.
>Is there a tool to analyze the time consumption in general? It will be
>a great news to know it.
>There is another thing to explain in the above picture. It seems that
>the Capacity is high as 1.617, but there are 64 bolts, most Capacity of
>it is low, as picture below shows.
>So, another puzzle is both the Execute latency and the Executed is
>about equal, but the Capacity turns out to be so much different.  Any
>The follow is another topology.
>Maybe the history_Put bolt has both high Capacity and larger Execute
>latency, this would definitely lead to the Complete latency as 56964ms?
>Thank you all for your time.
>发件人: 李波<>
>发送时间: 2017-08-07 16:56
>主题: RE: How to Improve Storm Application's Throughput
>1、Kafka PartitionNumber and  KafkaSpout’s parallelism
>2、Bolt’s business logical and Bolt’s parallelism
>是否开启了反压机制Config.TOPOLOGY_BACKPRESSURE_ENABLE (好像是storm1.0.0以后才有的)
>5、Executor‘s throughput params
>// Net io set
>config.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 1024 * 16); // default
>is 1024
>config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 1024 * 16);//
>batched; default is 1024
>config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 1024 * 16); //
>individual tuples; default is 1024
>李波 13813887096<>
>亚信是我家 发展靠大家
>From: 王 纯超 []
>Sent: 2017年8月7日 10:58
>To: user <>
>Cc: 姜艳春 <>;
>zhangxl_bds <>
>Subject: How to Improve Storm Application's Throughput
>I am now considering improve a Storm application's throughput because I
>find that the consumption speed of KafkaSpout is slower than the
>producing speed. And the lag gets larger and larger. Below is the bolt
>statistics. I tried to bring forward the tuple projection and filtering
>logic in a custom scheme with intention of reducing network traffic.
>However, after observation, things go contrary to my wishes. Am I going
>the wrong way? Are there any principles tuning Storm applications? Or
>could anyone give some suggestions for this specific case?

View raw message