spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Imran Rashid <iras...@cloudera.com>
Subject Re: java.lang.OutOfMemoryError: Unable to acquire bytes of memory
Date Thu, 14 Apr 2016 16:24:56 GMT
Hi Nezih,

I just reported a somewhat similar issue, and I have a potential fix --
SPARK-14560, looks like you are already watching it :).  You can try out
that patch, you have to explicitly enable the change in behavior with
"spark.shuffle.spillAfterRead=true".  Honestly, I don't think these issues
are the same, as I've always seen that case lead to acquiring 0 bytes,
while in your case you are requesting GBs and getting something pretty
close, so my hunch is that it is different ... but might be worth a shot to
see if it is the issue.

Turning on debug logging for TaskMemoryManager might help track the root
cause -- you'll get information on which consumers are using memory and
when there are spill attempts.  (Note that even if the patch I have for
SPARK-14560 doesn't fix your issue, it might still make those debug logs a
bit more clear, since it'll report memory used by Spillables.)

Imran

On Mon, Apr 4, 2016 at 10:52 PM, Nezih Yigitbasi <
nyigitbasi@netflix.com.invalid> wrote:

> Nope, I didn't have a chance to track the root cause, and IIRC we didn't
> observe it when dyn. alloc. is off.
>
> On Mon, Apr 4, 2016 at 6:16 PM Reynold Xin <rxin@databricks.com> wrote:
>
>> BTW do you still see this when dynamic allocation is off?
>>
>> On Mon, Apr 4, 2016 at 6:16 PM, Reynold Xin <rxin@databricks.com> wrote:
>>
>>> Nezih,
>>>
>>> Have you had a chance to figure out why this is happening?
>>>
>>>
>>> On Tue, Mar 22, 2016 at 1:32 AM, james <yiazhou@gmail.com> wrote:
>>>
>>>> I guess different workload cause diff result ?
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/java-lang-OutOfMemoryError-Unable-to-acquire-bytes-of-memory-tp16773p16789.html
>>>> Sent from the Apache Spark Developers List mailing list archive at
>>>> Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>
>>>>
>>>
>>

Mime
View raw message