spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Haberman <stephen.haber...@gmail.com>
Subject Re: gc/oome from 14,000 DiskBlockObjectWriters
Date Fri, 25 Oct 2013 15:30:15 GMT
> We're moving from Spark 0.7.x to Spark 0.8 (or master actually, to get
> the latest shuffle improvement)  

FWIW I went back from Spark master to Spark 0.8 and am seeing the same
behavior: OOMEs because of ~14,000 DiskBlockObjectWriters with 300k of
memory each.

So, it is not a change from the recent shuffle patch.

- Stephen


Mime
View raw message