storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kashyap Mhaisekar <kashya...@gmail.com>
Subject Re: Storm bolt instances - limits
Date Wed, 06 Jan 2016 01:03:26 GMT
Priyank,
Already seen this. My question was more on practical limitations on
parallelism than doing it right.

Thanks anyways.

Regards
Kashyap
On Jan 5, 2016 4:47 PM, "Priyank Shah" <pshah@hortonworks.com> wrote:

> Hi Kashyap,
>
> Please check out
> http://www.slideshare.net/ptgoetz/scaling-apache-storm-strata-hadoopworld-2014
>
>
> Priyank
>
>
> From: Kashyap Mhaisekar
> Reply-To: "user@storm.apache.org"
> Date: Tuesday, January 5, 2016 at 1:21 PM
> To: "user@storm.apache.org"
> Subject: Storm bolt instances - limits
>
> Hi,
> I had a question on the no. of instances of bolts that can be defined on a
> topology. Here is my use case -
> 1. 32 GB machine
> 2. 8 Processors
> 3. 12 Workers for each box (6700 to 6711)
> 4. Each worker has been defined with 2G heap and 64mb of XSS. Ulimit for
> storm is at 1024
>
> My topology has 5 bolts and a spout and If the combined parallelism is
> above 1000, i see OOM errrors indicating - java.lang.OutOfMemoryError:
> unable to create new native thread
>
> When this error occurs, topology is getting initialized and has not
> started processing.
>
> Is there a guideline from resource perspective as to how many instances
> can be created?
>
> Thanks
> Kashyap
>

Mime
View raw message