storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "" <>
Subject 回复: RE: How to Improve Storm Application's Throughput
Date Wed, 09 Aug 2017 03:00:03 GMT
Hi LiBo, Jungtaek :

Yes, storm tunning depends on situations. Thank u for your kindly advice.
The follow is one of my situations, any hints from you will be appreciated.
The topology just has a spout which reads message from kafka and a bolt to parse the message
and put it into the hbase.
As you can see from the above picture, the Execute latency of the bolt is small(0.5ms), but
the Complete latency is much more larger(4365ms), so as to slow down the throughput of the
Which part will consume so much additional time? the transfer between the spout and the bolt
? or the ack part? I tried to increase parallelism for the component, but it did not work.
Is there a tool to analyze the time consumption in general? It will be a great news to know

There is another thing to explain in the above picture. It seems that the Capacity is high
as 1.617, but there are 64 bolts, most Capacity of it is low, as picture below shows.
So, another puzzle is both the Execute latency and the Executed is about equal, but the Capacity
turns out to be so much different.  Any hints?

The follow is another topology.
Maybe the history_Put bolt has both high Capacity and larger Execute latency, this would definitely
lead to the Complete latency as 56964ms?

Thank you all for your time.


发件人: 李波
发送时间: 2017-08-07 16:56
抄送:; 'zhangxl_bds'
主题: RE: How to Improve Storm Application's Throughput
1、Kafka PartitionNumber and  KafkaSpout’s parallelism 
2、Bolt’s business logical and Bolt’s parallelism
是否开启了反压机制Config.TOPOLOGY_BACKPRESSURE_ENABLE (好像是storm1.0.0以后才有的)
5、Executor‘s throughput params
// Net io set
config.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 1024 * 16); // default is 1024
config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 1024 * 16);// batched; default is
config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 1024 * 16); // individual tuples; default
is 1024

李波 13813887096 北京亚信智慧数据科技有限公司
亚信是我家 发展靠大家
From: 王 纯超 [] 
Sent: 2017年8月7日 10:58
To: user <>
Cc: 姜艳春 <>; zhangxl_bds
Subject: How to Improve Storm Application's Throughput
    I am now considering improve a Storm application's throughput because I find that the
consumption speed of KafkaSpout is slower than the producing speed. And the lag gets larger
and larger. Below is the bolt statistics. I tried to bring forward the tuple projection and
filtering logic in a custom scheme with intention of reducing network traffic. However, after
observation, things go contrary to my wishes. Am I going the wrong way? Are there any principles
tuning Storm applications? Or could anyone give some suggestions for this specific case?
View raw message