spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Luthra <arun.lut...@gmail.com>
Subject Re: Workaround for spark 1.2.X roaringbitmap kryo problem?
Date Thu, 12 Mar 2015 18:58:52 GMT
The error is in the original post.

Here's the recipe that worked for me:
    kryo.register(Class.forName("org.roaringbitmap.RoaringArray$Element"))
    kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]])
    kryo.register(classOf[Array[Short]])
    kryo.register(classOf[org.roaringbitmap.RoaringBitmap])
    kryo.register(classOf[org.roaringbitmap.RoaringArray])
    kryo.register(classOf[org.roaringbitmap.ArrayContainer])

kryo.register(Class.forName("org.apache.spark.scheduler.HighlyCompressedMapStatus"))

So your Class.forName workaround worked, thanks!

On Thu, Mar 12, 2015 at 10:56 AM, Imran Rashid <irashid@cloudera.com> wrote:

> Giving a bit more detail on the error would make it a lot easier for
> others to help you out. Eg., in this case, it would have helped if included
> your actual compile error.
>
> In any case, I'm assuming your issue is b/c that class if private to
> spark.  You can sneak around that by using
> Class.forName("stringOfClassName") instead:
>
> scala> classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus]
>> <console>:8: error: class HighlyCompressedMapStatus in package scheduler
>> cannot be accessed in package org.apache.spark.scheduler
>>
>>   classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus]
>>                                                  ^
>> scala>
>> Class.forName("org.apache.spark.scheduler.HighlyCompressedMapStatus")
>> res1: Class[_] = class
>> org.apache.spark.scheduler.HighlyCompressedMapStatus
>
>
>
> hope this helps,
> Imran
>
>
> On Thu, Mar 12, 2015 at 12:47 PM, Arun Luthra <arun.luthra@gmail.com>
> wrote:
>
>> I'm using a pre-built Spark; I'm not trying to compile Spark.
>>
>> The compile error appears when I try to register
>> HighlyCompressedMapStatus in my program:
>>
>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
>>
>> If I don't register it, I get a runtime error saying that it needs to be
>> registered (the error is only when I turn on kryo).
>>
>> However the code is running smoothly with kryo turned off.
>>
>> On Wed, Mar 11, 2015 at 5:38 PM, Imran Rashid <irashid@cloudera.com>
>> wrote:
>>
>>> I'm not sure what you mean.   Are you asking how you can recompile all
>>> of spark and deploy it, instead of using one of the pre-built versions?
>>>
>>> https://spark.apache.org/docs/latest/building-spark.html
>>>
>>> Or are you seeing compile problems specifically w/
>>> HighlyCompressedMapStatus?   The code compiles fine, so I'm not sure what
>>> problem you are running into -- we'd need a lot more info to help
>>>
>>> On Tue, Mar 10, 2015 at 6:54 PM, Arun Luthra <arun.luthra@gmail.com>
>>> wrote:
>>>
>>>> Does anyone know how to get the HighlyCompressedMapStatus to compile?
>>>>
>>>> I will try turning off kryo in 1.2.0 and hope things don't break.  I
>>>> want to benefit from the MapOutputTracker fix in 1.2.0.
>>>>
>>>> On Tue, Mar 3, 2015 at 5:41 AM, Imran Rashid <irashid@cloudera.com>
>>>> wrote:
>>>>
>>>>> the scala syntax for arrays is Array[T], not T[], so you want to use
>>>>> something:
>>>>>
>>>>> kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]])
>>>>> kryo.register(classOf[Array[Short]])
>>>>>
>>>>> nonetheless, the spark should take care of this itself.  I'll look
>>>>> into it later today.
>>>>>
>>>>>
>>>>> On Mon, Mar 2, 2015 at 2:55 PM, Arun Luthra <arun.luthra@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I think this is a Java vs scala syntax issue. Will check.
>>>>>>
>>>>>> On Thu, Feb 26, 2015 at 8:17 PM, Arun Luthra <arun.luthra@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Problem is noted here:
>>>>>>> https://issues.apache.org/jira/browse/SPARK-5949
>>>>>>>
>>>>>>> I tried this as a workaround:
>>>>>>>
>>>>>>> import org.apache.spark.scheduler._
>>>>>>> import org.roaringbitmap._
>>>>>>>
>>>>>>> ...
>>>>>>>
>>>>>>>
>>>>>>> kryo.register(classOf[org.roaringbitmap.RoaringBitmap])
>>>>>>>     kryo.register(classOf[org.roaringbitmap.RoaringArray])
>>>>>>>     kryo.register(classOf[org.roaringbitmap.ArrayContainer])
>>>>>>>
>>>>>>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
>>>>>>>     kryo.register(classOf[org.roaringbitmap.RoaringArray$Element])
>>>>>>>     kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
>>>>>>>     kryo.register(classOf[short[]])
>>>>>>>
>>>>>>>
>>>>>>> in build file:
>>>>>>>
>>>>>>> libraryDependencies += "org.roaringbitmap" % "RoaringBitmap"
%
>>>>>>> "0.4.8"
>>>>>>>
>>>>>>>
>>>>>>> This fails to compile:
>>>>>>>
>>>>>>> ...:53: identifier expected but ']' found.
>>>>>>>
>>>>>>> [error]
>>>>>>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
>>>>>>>
>>>>>>> also:
>>>>>>>
>>>>>>> :54: identifier expected but ']' found.
>>>>>>>
>>>>>>> [error]     kryo.register(classOf[short[]])
>>>>>>> also:
>>>>>>>
>>>>>>> :51: class HighlyCompressedMapStatus in package scheduler cannot
be
>>>>>>> accessed in package org.apache.spark.scheduler
>>>>>>> [error]
>>>>>>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
>>>>>>>
>>>>>>>
>>>>>>> Suggestions?
>>>>>>>
>>>>>>> Arun
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message