The error is in the original post.

Here's the recipe that worked for me:
    kryo.register(Class.forName("org.roaringbitmap.RoaringArray$Element"))
    kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]])
    kryo.register(classOf[Array[Short]])  
    kryo.register(classOf[org.roaringbitmap.RoaringBitmap])
    kryo.register(classOf[org.roaringbitmap.RoaringArray])
    kryo.register(classOf[org.roaringbitmap.ArrayContainer])
    kryo.register(Class.forName("org.apache.spark.scheduler.HighlyCompressedMapStatus"))

So your Class.forName workaround worked, thanks!

On Thu, Mar 12, 2015 at 10:56 AM, Imran Rashid <irashid@cloudera.com> wrote:
Giving a bit more detail on the error would make it a lot easier for others to help you out. Eg., in this case, it would have helped if included your actual compile error.

In any case, I'm assuming your issue is b/c that class if private to spark.  You can sneak around that by using Class.forName("stringOfClassName") instead:

scala> classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus]
<console>:8: error: class HighlyCompressedMapStatus in package scheduler cannot be accessed in package org.apache.spark.scheduler
              classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus]
                                                 ^
scala> Class.forName("org.apache.spark.scheduler.HighlyCompressedMapStatus")
res1: Class[_] = class org.apache.spark.scheduler.HighlyCompressedMapStatus


hope this helps,
Imran
 

On Thu, Mar 12, 2015 at 12:47 PM, Arun Luthra <arun.luthra@gmail.com> wrote:
I'm using a pre-built Spark; I'm not trying to compile Spark.

The compile error appears when I try to register HighlyCompressedMapStatus in my program:
  kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])

If I don't register it, I get a runtime error saying that it needs to be registered (the error is only when I turn on kryo).

However the code is running smoothly with kryo turned off.

On Wed, Mar 11, 2015 at 5:38 PM, Imran Rashid <irashid@cloudera.com> wrote:
I'm not sure what you mean.   Are you asking how you can recompile all of spark and deploy it, instead of using one of the pre-built versions?


Or are you seeing compile problems specifically w/ HighlyCompressedMapStatus?   The code compiles fine, so I'm not sure what problem you are running into -- we'd need a lot more info to help

On Tue, Mar 10, 2015 at 6:54 PM, Arun Luthra <arun.luthra@gmail.com> wrote:
Does anyone know how to get the HighlyCompressedMapStatus to compile?

I will try turning off kryo in 1.2.0 and hope things don't break.  I want to benefit from the MapOutputTracker fix in 1.2.0.

On Tue, Mar 3, 2015 at 5:41 AM, Imran Rashid <irashid@cloudera.com> wrote:
the scala syntax for arrays is Array[T], not T[], so you want to use something:

kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]])
kryo.register(classOf[Array[Short]])

nonetheless, the spark should take care of this itself.  I'll look into it later today.


On Mon, Mar 2, 2015 at 2:55 PM, Arun Luthra <arun.luthra@gmail.com> wrote:
I think this is a Java vs scala syntax issue. Will check.

On Thu, Feb 26, 2015 at 8:17 PM, Arun Luthra <arun.luthra@gmail.com> wrote:

I tried this as a workaround:

import org.apache.spark.scheduler._
import org.roaringbitmap._

...


kryo.register(classOf[org.roaringbitmap.RoaringBitmap])
    kryo.register(classOf[org.roaringbitmap.RoaringArray])
    kryo.register(classOf[org.roaringbitmap.ArrayContainer])
    kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])
    kryo.register(classOf[org.roaringbitmap.RoaringArray$Element])
    kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])
    kryo.register(classOf[short[]])


in build file:

libraryDependencies += "org.roaringbitmap" % "RoaringBitmap" % "0.4.8"


This fails to compile:

...:53: identifier expected but ']' found.

[error]     kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])

also:

:54: identifier expected but ']' found.

[error]     kryo.register(classOf[short[]])

also:

:51: class HighlyCompressedMapStatus in package scheduler cannot be accessed in package org.apache.spark.scheduler
[error]     kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus])


Suggestions?

Arun