spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Malouf <malouf.g...@gmail.com>
Subject Re: Strange StackOverflowError with large amount of data in RDD
Date Sat, 07 Sep 2013 04:34:49 GMT
I did try the -Xss option, set it to 200m but that was little help.  These
are deserialize protobuf messages; is the best thing to always leave them
as byte arrays when mapping/filtering, etc until it's time to get results?
Or should this be done via Kryo?


On Fri, Sep 6, 2013 at 10:19 PM, Mike <spark@good-with-numbers.com> wrote:

> > Any idea why this would occur with a Join query?
>
> I'd guess not that you have a large amount of data causing it, but a
> deep object graph that you're deserializing.  Have you tried Kryo?  Have
> you tried the -Xss Java option?
>

Mime
View raw message