spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Luthra <arun.lut...@gmail.com>
Subject Re: Workaround for spark 1.2.X roaringbitmap kryo problem?
Date Tue, 10 Mar 2015 23:54:15 GMT
Does anyone know how to get the HighlyCompressedMapStatus to compile?

I will try turning off kryo in 1.2.0 and hope things don't break.  I want
to benefit from the MapOutputTracker fix in 1.2.0.

On Tue, Mar 3, 2015 at 5:41 AM, Imran Rashid <irashid@cloudera.com> wrote:

> the scala syntax for arrays is Array[T], not T[], so you want to use
> something:
>
> kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]])
> kryo.register(classOf[Array[Short]])
>
> nonetheless, the spark should take care of this itself.  I'll look into it
> later today.
>
>
> On Mon, Mar 2, 2015 at 2:55 PM, Arun Luthra <arun.luthra@gmail.com> wrote:
>
>> I think this is a Java vs scala syntax issue. Will check.
>>
>> On Thu, Feb 26, 2015 at 8:17 PM, Arun Luthra <arun.luthra@gmail.com>
>> wrote:
>>
>>> Problem is noted here: https://issues.apache.org/jira/browse/SPARK-5949
>>>
>>> I tried this as a workaround:
>>>
>>> import org.apache.spark.scheduler._
>>> import org.roaringbitmap._
>>>
>>> ...
>>>
>>>
>>> kryo.register(classOf[org.roaringbitmap.RoaringBitmap])
>>>     kryo.register(classOf[org.roaringbitmap.RoaringArray])
>>>     kryo.register(classOf[org.roaringbitmap.ArrayContainer])
>>>
>>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
>>>     kryo.register(classOf[org.roaringbitmap.RoaringArray$Element])
>>>     kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
>>>     kryo.register(classOf[short[]])
>>>
>>>
>>> in build file:
>>>
>>> libraryDependencies += "org.roaringbitmap" % "RoaringBitmap" % "0.4.8"
>>>
>>>
>>> This fails to compile:
>>>
>>> ...:53: identifier expected but ']' found.
>>>
>>> [error]
>>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
>>>
>>> also:
>>>
>>> :54: identifier expected but ']' found.
>>>
>>> [error]     kryo.register(classOf[short[]])
>>> also:
>>>
>>> :51: class HighlyCompressedMapStatus in package scheduler cannot be
>>> accessed in package org.apache.spark.scheduler
>>> [error]
>>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
>>>
>>>
>>> Suggestions?
>>>
>>> Arun
>>>
>>
>>
>

Mime
View raw message