spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Imran Rashid <iras...@cloudera.com>
Subject Re: Workaround for spark 1.2.X roaringbitmap kryo problem?
Date Tue, 03 Mar 2015 13:41:47 GMT
the scala syntax for arrays is Array[T], not T[], so you want to use
something:

kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]])
kryo.register(classOf[Array[Short]])

nonetheless, the spark should take care of this itself.  I'll look into it
later today.


On Mon, Mar 2, 2015 at 2:55 PM, Arun Luthra <arun.luthra@gmail.com> wrote:

> I think this is a Java vs scala syntax issue. Will check.
>
> On Thu, Feb 26, 2015 at 8:17 PM, Arun Luthra <arun.luthra@gmail.com>
> wrote:
>
>> Problem is noted here: https://issues.apache.org/jira/browse/SPARK-5949
>>
>> I tried this as a workaround:
>>
>> import org.apache.spark.scheduler._
>> import org.roaringbitmap._
>>
>> ...
>>
>>
>> kryo.register(classOf[org.roaringbitmap.RoaringBitmap])
>>     kryo.register(classOf[org.roaringbitmap.RoaringArray])
>>     kryo.register(classOf[org.roaringbitmap.ArrayContainer])
>>
>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
>>     kryo.register(classOf[org.roaringbitmap.RoaringArray$Element])
>>     kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
>>     kryo.register(classOf[short[]])
>>
>>
>> in build file:
>>
>> libraryDependencies += "org.roaringbitmap" % "RoaringBitmap" % "0.4.8"
>>
>>
>> This fails to compile:
>>
>> ...:53: identifier expected but ']' found.
>>
>> [error]
>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
>>
>> also:
>>
>> :54: identifier expected but ']' found.
>>
>> [error]     kryo.register(classOf[short[]])
>> also:
>>
>> :51: class HighlyCompressedMapStatus in package scheduler cannot be
>> accessed in package org.apache.spark.scheduler
>> [error]
>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
>>
>>
>> Suggestions?
>>
>> Arun
>>
>
>

Mime
View raw message