storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anishek Agarwal <anis...@gmail.com>
Subject Re: Storm bolt instances - limits
Date Wed, 06 Jan 2016 08:31:07 GMT
Hey Kashyap,

There seems to be a lot of threads that are created per bolt thread within
storm for processing. for ex we have apporx 100 parallelism per worker(with
all bolts in a topology) and we had to specify the -Xmx to 4 GB --
internally looked like the process was having abour 3.5-4K threads. The
number of instances that can be created will depend on the heap size per
worker gets so larger it is higher number of threads, from what you have
published it seems it would be better to have less workers per box say 4-8
and higher max heap per worker so you can create more threads per worker.

anishek

On Wed, Jan 6, 2016 at 6:33 AM, Kashyap Mhaisekar <kashyap.m@gmail.com>
wrote:

> Priyank,
> Already seen this. My question was more on practical limitations on
> parallelism than doing it right.
>
> Thanks anyways.
>
> Regards
> Kashyap
> On Jan 5, 2016 4:47 PM, "Priyank Shah" <pshah@hortonworks.com> wrote:
>
>> Hi Kashyap,
>>
>> Please check out
>> http://www.slideshare.net/ptgoetz/scaling-apache-storm-strata-hadoopworld-2014
>>
>>
>> Priyank
>>
>>
>> From: Kashyap Mhaisekar
>> Reply-To: "user@storm.apache.org"
>> Date: Tuesday, January 5, 2016 at 1:21 PM
>> To: "user@storm.apache.org"
>> Subject: Storm bolt instances - limits
>>
>> Hi,
>> I had a question on the no. of instances of bolts that can be defined on
>> a topology. Here is my use case -
>> 1. 32 GB machine
>> 2. 8 Processors
>> 3. 12 Workers for each box (6700 to 6711)
>> 4. Each worker has been defined with 2G heap and 64mb of XSS. Ulimit for
>> storm is at 1024
>>
>> My topology has 5 bolts and a spout and If the combined parallelism is
>> above 1000, i see OOM errrors indicating - java.lang.OutOfMemoryError:
>> unable to create new native thread
>>
>> When this error occurs, topology is getting initialized and has not
>> started processing.
>>
>> Is there a guideline from resource perspective as to how many instances
>> can be created?
>>
>> Thanks
>> Kashyap
>>
>

Mime
View raw message