spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Luthra <arun.lut...@gmail.com>
Subject Re: Workaround for spark 1.2.X roaringbitmap kryo problem?
Date Thu, 12 Mar 2015 17:47:03 GMT
I'm using a pre-built Spark; I'm not trying to compile Spark.

The compile error appears when I try to register HighlyCompressedMapStatus
in my program:

kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])

If I don't register it, I get a runtime error saying that it needs to be
registered (the error is only when I turn on kryo).

However the code is running smoothly with kryo turned off.

On Wed, Mar 11, 2015 at 5:38 PM, Imran Rashid <irashid@cloudera.com> wrote:

> I'm not sure what you mean.   Are you asking how you can recompile all of
> spark and deploy it, instead of using one of the pre-built versions?
>
> https://spark.apache.org/docs/latest/building-spark.html
>
> Or are you seeing compile problems specifically w/
> HighlyCompressedMapStatus?   The code compiles fine, so I'm not sure what
> problem you are running into -- we'd need a lot more info to help
>
> On Tue, Mar 10, 2015 at 6:54 PM, Arun Luthra <arun.luthra@gmail.com>
> wrote:
>
>> Does anyone know how to get the HighlyCompressedMapStatus to compile?
>>
>> I will try turning off kryo in 1.2.0 and hope things don't break.  I want
>> to benefit from the MapOutputTracker fix in 1.2.0.
>>
>> On Tue, Mar 3, 2015 at 5:41 AM, Imran Rashid <irashid@cloudera.com>
>> wrote:
>>
>>> the scala syntax for arrays is Array[T], not T[], so you want to use
>>> something:
>>>
>>> kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]])
>>> kryo.register(classOf[Array[Short]])
>>>
>>> nonetheless, the spark should take care of this itself.  I'll look into
>>> it later today.
>>>
>>>
>>> On Mon, Mar 2, 2015 at 2:55 PM, Arun Luthra <arun.luthra@gmail.com>
>>> wrote:
>>>
>>>> I think this is a Java vs scala syntax issue. Will check.
>>>>
>>>> On Thu, Feb 26, 2015 at 8:17 PM, Arun Luthra <arun.luthra@gmail.com>
>>>> wrote:
>>>>
>>>>> Problem is noted here:
>>>>> https://issues.apache.org/jira/browse/SPARK-5949
>>>>>
>>>>> I tried this as a workaround:
>>>>>
>>>>> import org.apache.spark.scheduler._
>>>>> import org.roaringbitmap._
>>>>>
>>>>> ...
>>>>>
>>>>>
>>>>> kryo.register(classOf[org.roaringbitmap.RoaringBitmap])
>>>>>     kryo.register(classOf[org.roaringbitmap.RoaringArray])
>>>>>     kryo.register(classOf[org.roaringbitmap.ArrayContainer])
>>>>>
>>>>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
>>>>>     kryo.register(classOf[org.roaringbitmap.RoaringArray$Element])
>>>>>     kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
>>>>>     kryo.register(classOf[short[]])
>>>>>
>>>>>
>>>>> in build file:
>>>>>
>>>>> libraryDependencies += "org.roaringbitmap" % "RoaringBitmap" % "0.4.8"
>>>>>
>>>>>
>>>>> This fails to compile:
>>>>>
>>>>> ...:53: identifier expected but ']' found.
>>>>>
>>>>> [error]
>>>>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
>>>>>
>>>>> also:
>>>>>
>>>>> :54: identifier expected but ']' found.
>>>>>
>>>>> [error]     kryo.register(classOf[short[]])
>>>>> also:
>>>>>
>>>>> :51: class HighlyCompressedMapStatus in package scheduler cannot be
>>>>> accessed in package org.apache.spark.scheduler
>>>>> [error]
>>>>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
>>>>>
>>>>>
>>>>> Suggestions?
>>>>>
>>>>> Arun
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message