spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com>
Subject Re: java.lang.OutOfMemoryError: unable to create new native thread
Date Wed, 25 Mar 2015 17:51:47 GMT
I have a YARN cluster where the max memory allowed is 16GB. I set 12G for
my driver, however i see OutOFMemory error even for this program
http://spark.apache.org/docs/1.3.0/sql-programming-guide.html#hive-tables .
What do you suggest ?

On Wed, Mar 25, 2015 at 8:23 AM, Thomas Gerber <thomas.gerber@radius.com>
wrote:

> So,
>
> 1. I reduced my  -XX:ThreadStackSize to 5m (instead of 10m - default is
> 1m), which is still OK for my need.
> 2. I reduced the executor memory to 44GB for a 60GB machine (instead of
> 49GB).
>
> This seems to have helped. Thanks to Matthew and Sean.
>
> Thomas
>
> On Tue, Mar 24, 2015 at 3:49 PM, Matt Silvey <matt.silvey@videoamp.com>
> wrote:
>
>> My memory is hazy on this but aren't there "hidden" limitations to
>> Linux-based threads?  I ran into some issues a couple of years ago where,
>> and here is the fuzzy part, the kernel wants to reserve virtual memory per
>> thread equal to the stack size.  When the total amount of reserved memory
>> (not necessarily resident memory) exceeds the memory of the system it
>> throws an OOM.  I'm looking for material to back this up.  Sorry for the
>> initial vague response.
>>
>> Matthew
>>
>> On Tue, Mar 24, 2015 at 12:53 PM, Thomas Gerber <thomas.gerber@radius.com
>> > wrote:
>>
>>> Additional notes:
>>> I did not find anything wrong with the number of threads (ps -u USER -L
>>> | wc -l): around 780 on the master and 400 on executors. I am running on
>>> 100 r3.2xlarge.
>>>
>>> On Tue, Mar 24, 2015 at 12:38 PM, Thomas Gerber <
>>> thomas.gerber@radius.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I am seeing various crashes in spark on large jobs which all share a
>>>> similar exception:
>>>>
>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>> at java.lang.Thread.start0(Native Method)
>>>> at java.lang.Thread.start(Thread.java:714)
>>>>
>>>> I increased nproc (i.e. ulimit -u) 10 fold, but it doesn't help.
>>>>
>>>> Does anyone know how to avoid those kinds of errors?
>>>>
>>>> Noteworthy: I added -XX:ThreadStackSize=10m on both driver and executor
>>>> extra java options, which might have amplified the problem.
>>>>
>>>> Thanks for you help,
>>>> Thomas
>>>>
>>>
>>>
>>
>


-- 
Deepak

Mime
View raw message