spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Imran Rashid <iras...@cloudera.com>
Subject Re: com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index:
Date Tue, 05 May 2015 14:58:00 GMT
Are you setting a really large max buffer size for kryo?
Was this fixed by https://issues.apache.org/jira/browse/SPARK-6405 ?


If not, we should open up another issue to get a better warning in these
cases.

On Tue, May 5, 2015 at 2:47 AM, shahab <shahab.mokari@gmail.com> wrote:

> Thanks Tristan for sharing this. Actually this happens when I am reading a
> csv file of 3.5 GB.
>
> best,
> /Shahab
>
>
>
> On Tue, May 5, 2015 at 9:15 AM, Tristan Blakers <tristan@blackfrog.org>
> wrote:
>
>> Hi Shahab,
>>
>> I’ve seen exceptions very similar to this (it also manifests as negative
>> array size exception), and I believe it’s a really bug in Kryo.
>>
>> See this thread:
>>
>> http://mail-archives.us.apache.org/mod_mbox/spark-user/201502.mbox/%3CCAG02iJuw3Oqbi2t8acb5NLRvXsO2xmas1qRQd_4fq1TgvvJMFg@mail.gmail.com%3E
>>
>> Manifests in all of the following situations when working with an object
>> graph in excess of a few GB: Joins, Broadcasts, and when using the hadoop
>> save APIs.
>>
>> Tristan
>>
>>
>> On 3 May 2015 at 07:26, Olivier Girardot <ssaboum@gmail.com> wrote:
>>
>>> Can you post your code, otherwise there's not much we can do.
>>>
>>> Regards,
>>>
>>> Olivier.
>>>
>>> Le sam. 2 mai 2015 à 21:15, shahab <shahab.mokari@gmail.com> a écrit
:
>>>
>>>> Hi,
>>>>
>>>> I am using sprak-1.2.0 and I used Kryo serialization but I get the
>>>> following excepton.
>>>>
>>>> java.io.IOException: com.esotericsoftware.kryo.KryoException:
>>>> java.lang.IndexOutOfBoundsException: Index: 3448, Size: 1
>>>>
>>>> I do apprecciate if anyone could tell me how I can resolve this?
>>>>
>>>> best,
>>>> /Shahab
>>>>
>>>
>>
>

Mime
View raw message