storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bert Corderman <bertc...@gmail.com>
Subject Re: Storm Scaling Issues
Date Fri, 16 May 2014 07:19:28 GMT
how many ackers?  make sure this is set to number of workers.

what do you have for max spout pending?  we configured to have  5-10
seconds of messages in pending state.  (divide 5-10 seconds by avg tuple
processing time)

Bert


On Wed, May 14, 2014 at 4:35 PM, Lasantha Fernando
<lasantha.fdo@gmail.com>wrote:

> Hi Nathan,
>
> Tried increasing the number of workers with no luck. Also increased the
> parallelism of the throughput measuring bolt and checked the aggregate
> throughput, but that does not seem to be the bottleneck. Also the capacity
> value of the throughput measuring bolt in Storm UI is at around ~0.12.
>
> Will try out more configurations and see. Thank you very much for your
> tips.
>
> Any other tweaks I might try out?
>
> Thanks,
> Lasantha
>
>
> On Tue, May 13, 2014 at 6:38 PM, Nathan Leung <ncleung@gmail.com> wrote:
>
>> For 20 spouts and even number of processing bolts 3 seems like an odd
>> number of workers. Also are you sure you're not bottlenecked by your
>> throughput measuring bolt?
>> On May 13, 2014 2:43 AM, "Lasantha Fernando" <lasantha.fdo@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> Is there any guide or hints on how to configure storm to scale better?
>>>
>>> I was running some tests with a custom scheduler and found that the
>>> throughput did not scale as expected. Any pointers on what I am doing wrong?
>>>
>>> Parallelism 2 4 8 16 Single Node (Avg) 166099 161539.5 193986 N/A Two
>>> Node (Avg) 160988 165563 174675.5 177624.5
>>> The topology is as follows.
>>>
>>> Spout (Generates events continuously) -> Processing Bolt -> Throughput
>>> Measurement Bolt
>>>
>>> Parallelism is varied for the processing bolt.
>>>
>>> Parallelism for spout and throughput measuring bolt is kept constant at
>>> 20 and 1 respectively.
>>>
>>> Topology.NUM_WORKERS = 3
>>>
>>> Custom scheduler code is available at [1]. Topology code is available at
>>> [2]. Any pointers would be much appreciated.
>>>
>>> Thanks,
>>> Lasantha
>>>
>>> [1]
>>> https://github.com/sajithshn/storm-schedulers/blob/master/src/main/java/org/wso2/siddhi/storm/scheduler/RoundRobinStormScheduler.java
>>> [2]
>>> https://github.com/lasanthafdo/siddhi-storm/blob/master/src/main/java/org/wso2/siddhi/storm/StockDataTopology.java
>>>
>>
>

Mime
View raw message