spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <>
Subject Re: OutOfMemory in "cogroup"
Date Mon, 27 Oct 2014 16:13:21 GMT
On Monday, October 27, 2014, Shixiong Zhu <> wrote:

> We encountered some special OOM cases of "cogroup" when the data in one
> partition is not balanced.
> 1. The estimated size of used memory is inaccurate. For example, there are
> too many values for some special keys. Because SizeEstimator.visitArray
> only samples at most 100 cells for an array, it may miss most of these
> special keys and get a wrong estimated size of used memory. In our case, it
> reports a CompactBuffer is only 27M, but actually it's more than 5G. Since
> the estimated value is wrong, the CompactBuffer won't be spilled and cause
> OOM.
You can change spark.shuffle.safetyFraction , but that is a really big
margin to add.

> 2. There are too many values for a special key and these values cannot fit
> into memory. Spilling data to disk helps nothing because cogroup needs to
> read all values for a key into memory.
> What are you doing with the data after you co-group it? Are you doing a
reduction? You could try doing a union and a reduceByKey

> Any suggestion to solve these OOM cases? Thank you,.
> Best Regards,
> Shixiong Zhu

View raw message