spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: SparkException: Expect only DirectTaskResults when using localScheduler()
Date Sat, 18 Jan 2014 18:47:51 GMT
It's 10 MB:

http://spark.incubator.apache.org/docs/latest/configuration.html

- Patrick

On Sat, Jan 18, 2014 at 12:08 AM, Nick Pentreath
<nick.pentreath@gmail.com> wrote:
> Ok that makes sense thanks. What is the default all a frame size?
> —
> Sent from Mailbox for iPhone
>
>
> On Sat, Jan 18, 2014 at 9:54 AM, Patrick Wendell <pwendell@gmail.com> wrote:
>>
>> This is a bug that was fixed and will be part of 0.8.2:
>>
>> https://github.com/apache/incubator-spark/pull/281
>>
>> A workaround is setting the akka frame size to be larger using
>> spark.akka.frameSize.
>>
>> The issue is that we added a mechanism to circumvent akka for large
>> task results, but the implementation assumed this would never happen
>> in local mode. That assumption turned out to be wrong, so it was
>> patched in #281.
>>
>> - Patrick
>>
>> On Fri, Jan 17, 2014 at 9:53 PM, Nick Pentreath
>> <nick.pentreath@gmail.com> wrote:
>> > I'm also running into this issue. If it is related to a setting then the
>> > error message should surely be cleaned up a bit to be more helpful in
>> > suggesting a fix?
>> > —
>> > Sent from Mailbox for iPhone
>> >
>> >
>> > On Sat, Jan 18, 2014 at 2:35 AM, Hussam_Jarada@dell.com
>> > <Hussam_Jarada@dell.com> wrote:
>> >>
>> >> Hi,
>> >>
>> >>
>> >>
>> >> Can someone please explain what it means the following exception
>> >> ‘SparkException: Expect only DirectTaskResults when using
>> >> localScheduler()’?
>> >>
>> >> This is related to spark.akka.frameSize?
>> >>
>> >> I am running single node local spark 0.8.1
>> >>
>> >>
>> >>
>> >> Thanks,
>> >>
>> >> Hussam
>> >
>> >
>
>

Mime
View raw message