storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kashyap Mhaisekar <kashya...@gmail.com>
Subject Re: Storm bolt instances - limits
Date Thu, 07 Jan 2016 13:13:10 GMT
That's one area that is confusing... Xss indicates stack size at each
thread level. Increasing it means that each thread effectively has more
memory for stack and hence overall requirements for memory increase. Am not
really sure how increasing stack size allows for more thread creation. Will
test this part and confirm back

Thanks
Kashyap
On Jan 7, 2016 02:58, "Anishek Agarwal" <anishek@gmail.com> wrote:

> Yeh I have encountered the same error, and as you said its because there
> is a limit to the number of threads with a specific stack size, that can be
> created for a given heap,  there are a few discussions on Stackoverflow for
> the same , I also tried to verify this via a standalone program but it was
> behaving very non intuitively on the local machine, for ex it was stopping
> after creating 980 threads always with the same stack size of 256k and Heap
> size varying from default to 1 GB.
>
> we are using oracle jdk and hence in our case the the stack size cannont
> below 228k we have it as 256k with about 4 GB of -Xmx its working fine with
> increased number of threads.
>
> On Wed, Jan 6, 2016 at 7:14 PM, Kashyap Mhaisekar <kashyap.m@gmail.com>
> wrote:
>
>> Thanks Anishek.
>> I read that OOM due to lack of native threads is because of stack size or
>> because of OS running out of threads.
>>
>> Did you specifically encounter this error in your use case?
>>
>> Thanks
>> Kashyap
>> On Jan 6, 2016 02:32, "Anishek Agarwal" <anishek@gmail.com> wrote:
>>
>>> Hey Kashyap,
>>>
>>> There seems to be a lot of threads that are created per bolt thread
>>> within storm for processing. for ex we have apporx 100 parallelism per
>>> worker(with all bolts in a topology) and we had to specify the -Xmx to 4 GB
>>> -- internally looked like the process was having abour 3.5-4K threads. The
>>> number of instances that can be created will depend on the heap size per
>>> worker gets so larger it is higher number of threads, from what you have
>>> published it seems it would be better to have less workers per box say 4-8
>>> and higher max heap per worker so you can create more threads per worker.
>>>
>>> anishek
>>>
>>> On Wed, Jan 6, 2016 at 6:33 AM, Kashyap Mhaisekar <kashyap.m@gmail.com>
>>> wrote:
>>>
>>>> Priyank,
>>>> Already seen this. My question was more on practical limitations on
>>>> parallelism than doing it right.
>>>>
>>>> Thanks anyways.
>>>>
>>>> Regards
>>>> Kashyap
>>>> On Jan 5, 2016 4:47 PM, "Priyank Shah" <pshah@hortonworks.com> wrote:
>>>>
>>>>> Hi Kashyap,
>>>>>
>>>>> Please check out
>>>>> http://www.slideshare.net/ptgoetz/scaling-apache-storm-strata-hadoopworld-2014
>>>>>
>>>>>
>>>>> Priyank
>>>>>
>>>>>
>>>>> From: Kashyap Mhaisekar
>>>>> Reply-To: "user@storm.apache.org"
>>>>> Date: Tuesday, January 5, 2016 at 1:21 PM
>>>>> To: "user@storm.apache.org"
>>>>> Subject: Storm bolt instances - limits
>>>>>
>>>>> Hi,
>>>>> I had a question on the no. of instances of bolts that can be defined
>>>>> on a topology. Here is my use case -
>>>>> 1. 32 GB machine
>>>>> 2. 8 Processors
>>>>> 3. 12 Workers for each box (6700 to 6711)
>>>>> 4. Each worker has been defined with 2G heap and 64mb of XSS. Ulimit
>>>>> for storm is at 1024
>>>>>
>>>>> My topology has 5 bolts and a spout and If the combined parallelism is
>>>>> above 1000, i see OOM errrors indicating - java.lang.OutOfMemoryError:
>>>>> unable to create new native thread
>>>>>
>>>>> When this error occurs, topology is getting initialized and has not
>>>>> started processing.
>>>>>
>>>>> Is there a guideline from resource perspective as to how many
>>>>> instances can be created?
>>>>>
>>>>> Thanks
>>>>> Kashyap
>>>>>
>>>>
>>>
>

Mime
View raw message