spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish Rangole <arang...@gmail.com>
Subject Re: oome from blockmanager
Date Tue, 05 Nov 2013 18:23:57 GMT
Awesome indeed! Is this available in the main branch now?

Thanks!
On Nov 5, 2013 9:50 AM, "Stephen Haberman" <stephen.haberman@gmail.com>
wrote:

>
> > As a followup on this, the memory footprint of all shuffle metadata
> > has been  greatly reduced. For your original workload with 7k
> > mappers, 7k reducers, and 5 machines, the total metadata size should
> > have decreased from ~3.3 GB to ~80 MB.
>
> Wow! Awesome work, Aaron! Thanks for the amazing quick follow up.
>
> - Stephen
>
>

Mime
View raw message