spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Haberman <stephen.haber...@gmail.com>
Subject Re: oome from blockmanager
Date Tue, 05 Nov 2013 16:50:22 GMT

> As a followup on this, the memory footprint of all shuffle metadata
> has been  greatly reduced. For your original workload with 7k
> mappers, 7k reducers, and 5 machines, the total metadata size should
> have decreased from ~3.3 GB to ~80 MB.

Wow! Awesome work, Aaron! Thanks for the amazing quick follow up.

- Stephen


Mime
View raw message