storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Priyank Shah <ps...@hortonworks.com>
Subject Re: Storm bolt instances - limits
Date Tue, 05 Jan 2016 22:47:00 GMT
Hi Kashyap,

Please check out http://www.slideshare.net/ptgoetz/scaling-apache-storm-strata-hadoopworld-2014

Priyank


From: Kashyap Mhaisekar
Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>"
Date: Tuesday, January 5, 2016 at 1:21 PM
To: "user@storm.apache.org<mailto:user@storm.apache.org>"
Subject: Storm bolt instances - limits

Hi,
I had a question on the no. of instances of bolts that can be defined on a topology. Here
is my use case -
1. 32 GB machine
2. 8 Processors
3. 12 Workers for each box (6700 to 6711)
4. Each worker has been defined with 2G heap and 64mb of XSS. Ulimit for storm is at 1024

My topology has 5 bolts and a spout and If the combined parallelism is above 1000, i see OOM
errrors indicating - java.lang.OutOfMemoryError: unable to create new native thread

When this error occurs, topology is getting initialized and has not started processing.

Is there a guideline from resource perspective as to how many instances can be created?

Thanks
Kashyap
Mime
View raw message