spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tristan Blakers <tris...@blackfrog.org>
Subject Re: com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index:
Date Wed, 06 May 2015 07:29:29 GMT
Hi Imran,

I had tried setting a really huge kryo buffer size (GB), but it didn’t make
any difference.

In my data sets, objects are no more than 1KB each, and don’t form a graph,
so I don’t think the buffer size should need to be larger than a few MB,
except perhaps for reasons of efficiency?

The exception usually occurs in
“com.esotericsoftware.kryo.util.IdentityObjectIntMap”
when it is resizing (or a similar operation), implying there are too many
object references, though it’s hard to see how I could get to 2b references
from a few million objects...

T

On 6 May 2015 at 00:58, Imran Rashid <irashid@cloudera.com> wrote:

> Are you setting a really large max buffer size for kryo?
> Was this fixed by https://issues.apache.org/jira/browse/SPARK-6405 ?
>
>
> If not, we should open up another issue to get a better warning in these
> cases.
>
> On Tue, May 5, 2015 at 2:47 AM, shahab <shahab.mokari@gmail.com> wrote:
>
>> Thanks Tristan for sharing this. Actually this happens when I am reading
>> a csv file of 3.5 GB.
>>
>> best,
>> /Shahab
>>
>>
>>
>> On Tue, May 5, 2015 at 9:15 AM, Tristan Blakers <tristan@blackfrog.org>
>> wrote:
>>
>>> Hi Shahab,
>>>
>>> I’ve seen exceptions very similar to this (it also manifests as negative
>>> array size exception), and I believe it’s a really bug in Kryo.
>>>
>>> See this thread:
>>>
>>> http://mail-archives.us.apache.org/mod_mbox/spark-user/201502.mbox/%3CCAG02iJuw3Oqbi2t8acb5NLRvXsO2xmas1qRQd_4fq1TgvvJMFg@mail.gmail.com%3E
>>>
>>> Manifests in all of the following situations when working with an object
>>> graph in excess of a few GB: Joins, Broadcasts, and when using the hadoop
>>> save APIs.
>>>
>>> Tristan
>>>
>>>
>>> On 3 May 2015 at 07:26, Olivier Girardot <ssaboum@gmail.com> wrote:
>>>
>>>> Can you post your code, otherwise there's not much we can do.
>>>>
>>>> Regards,
>>>>
>>>> Olivier.
>>>>
>>>> Le sam. 2 mai 2015 à 21:15, shahab <shahab.mokari@gmail.com> a écrit
:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am using sprak-1.2.0 and I used Kryo serialization but I get the
>>>>> following excepton.
>>>>>
>>>>> java.io.IOException: com.esotericsoftware.kryo.KryoException:
>>>>> java.lang.IndexOutOfBoundsException: Index: 3448, Size: 1
>>>>>
>>>>> I do apprecciate if anyone could tell me how I can resolve this?
>>>>>
>>>>> best,
>>>>> /Shahab
>>>>>
>>>>
>>>
>>
>

Mime
View raw message