Already seen this. My question was more on practical limitations on parallelism than doing it right.
From: Kashyap Mhaisekar
Date: Tuesday, January 5, 2016 at 1:21 PM
Subject: Storm bolt instances - limits
Hi,I had a question on the no. of instances of bolts that can be defined on a topology. Here is my use case -1. 32 GB machine2. 8 Processors3. 12 Workers for each box (6700 to 6711)4. Each worker has been defined with 2G heap and 64mb of XSS. Ulimit for storm is at 1024
My topology has 5 bolts and a spout and If the combined parallelism is above 1000, i see OOM errrors indicating - java.lang.OutOfMemoryError: unable to create new native thread
When this error occurs, topology is getting initialized and has not started processing.
Is there a guideline from resource perspective as to how many instances can be created?