spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Davidson <ilike...@gmail.com>
Subject Re: oome from blockmanager
Date Sun, 01 Dec 2013 21:58:11 GMT
Thanks for the update, and glad to hear things are working well!


On Sun, Dec 1, 2013 at 1:56 PM, Stephen Haberman <stephen.haberman@gmail.com
> wrote:

>
> > The short term solutions have already been discussed: decrease the
> > number of reducers (and mappers, if you need them to be tied) or
> > potentially turn off compression if Snappy is holding too much buffer
> > space.
>
> Just to follow up with this (sorry for the delay; I was busy/out for
> Thanksgiving), but after chatting about it, we've moved from m1.larges
> to m1.xlarges, and nudged our partition size up (and so partition
> number down) and things are going quite well now.
>
> So, if we had a 40-machine m1.large job, we're now running it with 20
> m1.xlarges, and as expected it takes basically the same time/cost
> (since m1.xlarges have basically twice the resources), which is great.
>
> I'm still tempted to play around with turning off compression, but this is
> no longer a squeaky wheel, so we probably won't actively investigate it.
>
> Thanks for all the help, Aaron/Patrick/the list, we really appreciate
> it.
>
> - Stephen
>
>
>
>

Mime
View raw message