From user-return-28599-apmail-spark-user-archive=spark.apache.org@spark.apache.org Thu Mar 12 18:59:22 2015 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8686A17D73 for ; Thu, 12 Mar 2015 18:59:22 +0000 (UTC) Received: (qmail 25043 invoked by uid 500); 12 Mar 2015 18:59:19 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 24966 invoked by uid 500); 12 Mar 2015 18:59:19 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 24947 invoked by uid 99); 12 Mar 2015 18:59:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Mar 2015 18:59:19 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of arun.luthra@gmail.com designates 74.125.82.42 as permitted sender) Received: from [74.125.82.42] (HELO mail-wg0-f42.google.com) (74.125.82.42) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Mar 2015 18:58:54 +0000 Received: by wggx13 with SMTP id x13so18475631wgg.4 for ; Thu, 12 Mar 2015 11:58:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=eDHDOpzb+s2iDkAShPO3jc7qIM37eL/iDaxfFV7jYhc=; b=QG/fF2CpiSp7DyPMwKZ4NUs+MXykZIkheHfOeHZS/Su90oHr6tQ1Esc+L+Lv4E+tPp ACQiLAPkgpmGuWYG88DwGV7Lx8+rAabAN9F8nCTTUNBo9FHCcznIgSrH//UJHNwezcOc NIMIFOO1q0qeivjvCX822k720alp8FPOAE3/D7S0LixI8pJBj80nO9v2AMQeVfBg//E/ 4TwP3VBQnXr0QWs16Zeu9gyyFnNpvhZEsc0HieCZMy2iTYfrNf72HIzZicoRUHU51xEA yUdZOGeYpkzZiFesJo5Ru/U9Xx7AQz/6K++2U6lmZfdPt7n6ukaRld7OHOFlv6SxJeQL 7zTA== MIME-Version: 1.0 X-Received: by 10.194.187.236 with SMTP id fv12mr91161070wjc.131.1426186732829; Thu, 12 Mar 2015 11:58:52 -0700 (PDT) Received: by 10.28.146.21 with HTTP; Thu, 12 Mar 2015 11:58:52 -0700 (PDT) In-Reply-To: References: Date: Thu, 12 Mar 2015 11:58:52 -0700 Message-ID: Subject: Re: Workaround for spark 1.2.X roaringbitmap kryo problem? From: Arun Luthra To: Imran Rashid Cc: "user@spark.apache.org" Content-Type: multipart/alternative; boundary=047d7bea40cc98003f05111bf99d X-Virus-Checked: Checked by ClamAV on apache.org --047d7bea40cc98003f05111bf99d Content-Type: text/plain; charset=UTF-8 The error is in the original post. Here's the recipe that worked for me: kryo.register(Class.forName("org.roaringbitmap.RoaringArray$Element")) kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]]) kryo.register(classOf[Array[Short]]) kryo.register(classOf[org.roaringbitmap.RoaringBitmap]) kryo.register(classOf[org.roaringbitmap.RoaringArray]) kryo.register(classOf[org.roaringbitmap.ArrayContainer]) kryo.register(Class.forName("org.apache.spark.scheduler.HighlyCompressedMapStatus")) So your Class.forName workaround worked, thanks! On Thu, Mar 12, 2015 at 10:56 AM, Imran Rashid wrote: > Giving a bit more detail on the error would make it a lot easier for > others to help you out. Eg., in this case, it would have helped if included > your actual compile error. > > In any case, I'm assuming your issue is b/c that class if private to > spark. You can sneak around that by using > Class.forName("stringOfClassName") instead: > > scala> classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus] >> :8: error: class HighlyCompressedMapStatus in package scheduler >> cannot be accessed in package org.apache.spark.scheduler >> >> classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus] >> ^ >> scala> >> Class.forName("org.apache.spark.scheduler.HighlyCompressedMapStatus") >> res1: Class[_] = class >> org.apache.spark.scheduler.HighlyCompressedMapStatus > > > > hope this helps, > Imran > > > On Thu, Mar 12, 2015 at 12:47 PM, Arun Luthra > wrote: > >> I'm using a pre-built Spark; I'm not trying to compile Spark. >> >> The compile error appears when I try to register >> HighlyCompressedMapStatus in my program: >> >> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus]) >> >> If I don't register it, I get a runtime error saying that it needs to be >> registered (the error is only when I turn on kryo). >> >> However the code is running smoothly with kryo turned off. >> >> On Wed, Mar 11, 2015 at 5:38 PM, Imran Rashid >> wrote: >> >>> I'm not sure what you mean. Are you asking how you can recompile all >>> of spark and deploy it, instead of using one of the pre-built versions? >>> >>> https://spark.apache.org/docs/latest/building-spark.html >>> >>> Or are you seeing compile problems specifically w/ >>> HighlyCompressedMapStatus? The code compiles fine, so I'm not sure what >>> problem you are running into -- we'd need a lot more info to help >>> >>> On Tue, Mar 10, 2015 at 6:54 PM, Arun Luthra >>> wrote: >>> >>>> Does anyone know how to get the HighlyCompressedMapStatus to compile? >>>> >>>> I will try turning off kryo in 1.2.0 and hope things don't break. I >>>> want to benefit from the MapOutputTracker fix in 1.2.0. >>>> >>>> On Tue, Mar 3, 2015 at 5:41 AM, Imran Rashid >>>> wrote: >>>> >>>>> the scala syntax for arrays is Array[T], not T[], so you want to use >>>>> something: >>>>> >>>>> kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]]) >>>>> kryo.register(classOf[Array[Short]]) >>>>> >>>>> nonetheless, the spark should take care of this itself. I'll look >>>>> into it later today. >>>>> >>>>> >>>>> On Mon, Mar 2, 2015 at 2:55 PM, Arun Luthra >>>>> wrote: >>>>> >>>>>> I think this is a Java vs scala syntax issue. Will check. >>>>>> >>>>>> On Thu, Feb 26, 2015 at 8:17 PM, Arun Luthra >>>>>> wrote: >>>>>> >>>>>>> Problem is noted here: >>>>>>> https://issues.apache.org/jira/browse/SPARK-5949 >>>>>>> >>>>>>> I tried this as a workaround: >>>>>>> >>>>>>> import org.apache.spark.scheduler._ >>>>>>> import org.roaringbitmap._ >>>>>>> >>>>>>> ... >>>>>>> >>>>>>> >>>>>>> kryo.register(classOf[org.roaringbitmap.RoaringBitmap]) >>>>>>> kryo.register(classOf[org.roaringbitmap.RoaringArray]) >>>>>>> kryo.register(classOf[org.roaringbitmap.ArrayContainer]) >>>>>>> >>>>>>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus]) >>>>>>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element]) >>>>>>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]]) >>>>>>> kryo.register(classOf[short[]]) >>>>>>> >>>>>>> >>>>>>> in build file: >>>>>>> >>>>>>> libraryDependencies += "org.roaringbitmap" % "RoaringBitmap" % >>>>>>> "0.4.8" >>>>>>> >>>>>>> >>>>>>> This fails to compile: >>>>>>> >>>>>>> ...:53: identifier expected but ']' found. >>>>>>> >>>>>>> [error] >>>>>>> kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]]) >>>>>>> >>>>>>> also: >>>>>>> >>>>>>> :54: identifier expected but ']' found. >>>>>>> >>>>>>> [error] kryo.register(classOf[short[]]) >>>>>>> also: >>>>>>> >>>>>>> :51: class HighlyCompressedMapStatus in package scheduler cannot be >>>>>>> accessed in package org.apache.spark.scheduler >>>>>>> [error] >>>>>>> kryo.register(classOf[org.apache.spark.scheduler.HighlyCompressedMapStatus]) >>>>>>> >>>>>>> >>>>>>> Suggestions? >>>>>>> >>>>>>> Arun >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> > --047d7bea40cc98003f05111bf99d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
The error is in the original post.

Here= 's the recipe that worked for me:
=C2=A0 =C2=A0 kryo.reg= ister(Class.forName("org.roaringbitmap.RoaringArray$Element"))
=C2=A0 =C2=A0 kryo.register(classOf[Array[org.roaringbitmap.Roaring= Array$Element]])
=C2=A0 =C2=A0 kryo.register(classOf[Array[Short]= ]) =C2=A0
=C2=A0 =C2=A0 kryo.register(classOf[org.roaringbitmap.R= oaringBitmap])
=C2=A0 =C2=A0 kryo.register(classOf[org.roaringbit= map.RoaringArray])
=C2=A0 =C2=A0 kryo.register(classOf[org.roarin= gbitmap.ArrayContainer])
=C2=A0 =C2=A0 kryo.register(Class.forNam= e("org.apache.spark.scheduler.HighlyCompressedMapStatus"))
<= /div>

So your Class.forName workaround worked, thanks!

On Thu,= Mar 12, 2015 at 10:56 AM, Imran Rashid <irashid@cloudera.com> wrote:
Giving a bi= t more detail on the error would make it a lot easier for others to help yo= u out. Eg., in this case, it would have helped if included your actual comp= ile error.

In any case, I'm assuming your issue is b= /c that class if private to spark.=C2=A0 You can sneak around that by using= Class.forName("stringOfClassName") instead:

=
scala> classOf[org.apache.spark.scheduler.HighlyCo= mpressedMapStatus]
<console>:8: error: class HighlyCompressedMapSt= atus in package scheduler cannot be accessed in package org.apache.spark.sc= heduler
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0classOf[or= g.apache.spark.scheduler.HighlyCompressedMapStatus]
=C2=A0=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0=C2=A0^
scala> Class.forName("org.apache.spark.scheduler.H= ighlyCompressedMapStatus")
res1: Class[_] =3D class org.apache.spar= k.scheduler.HighlyCompressedMapStatus


<= /div>
hope this helps,
Imran
=C2=A0

On Thu, Mar 12, 2015 at 12:47 PM, Arun Luthra <arun.luthra@gmail.com> wrote:
I&= #39;m using a pre-built Spark; I'm not trying to compile Spark.<= div style=3D"font-size:12.8000001907349px">
The compile error appears when I try to register High= lyCompressedMapStatus in my program:
=C2=A0 kryo.register(classOf[org.apache.spark.scheduler.HighlyCo= mpressedMapStatus])
If I don't register= it, I get a runtime error saying that it needs to be registered (the error= is only when I turn on kryo).

However the code= is running smoothly with kryo turned off.

On Wed, Mar 11, 2015 at 5= :38 PM, Imran Rashid <irashid@cloudera.com> wrote:
I'm not s= ure what you mean. =C2=A0 Are you asking how you can recompile all of spark= and deploy it, instead of using one of the pre-built versions?


Or are you seeing compile problems speci= fically w/ HighlyCompressedMapStatus? =C2=A0 The code compiles fine, so I&#= 39;m not sure what problem you are running into -- we'd need a lot more= info to help

On Tue, Mar 10, 2015 at 6:54 PM, Arun Luthra <arun.l= uthra@gmail.com> wrote:
Does anyone know how to get the HighlyCompressedMapStatus to = compile?

I will try turning off kryo in 1.2.0 and hope t= hings don't break.=C2=A0 I want to benefit from the MapOutputTracker fi= x in 1.2.0.

On Tue, Mar 3, 2015 at 5:41 AM, Imran Rashid <irashid@= cloudera.com> wrote:
the scala syntax for arrays is Array[T], not T[], so you want to= use something:

kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]])
kryo.register(classOf[Array= [Short]])

=
nonetheless, the spark should t= ake care of this itself.=C2=A0 I'll look into it later today.


On Mon, Mar 2, 2015 at 2:5= 5 PM, Arun Luthra <arun.luthra@gmail.com> wrote:
I think this is a Java vs scala= syntax issue. Will check.

On Thu, Feb 26, 2015 at 8:17 PM, Arun Luthra <arun.luthra@gmail.com> wrote:

I tried this as a w= orkaround:

import org.apache.spark.scheduler._=
import org.roaringbitmap._

...


kryo.register(classOf[org.roaringbitm= ap.RoaringBitmap])
=C2=A0 =C2=A0 kryo.register(classOf[org.roarin= gbitmap.RoaringArray])
=C2=A0 =C2=A0 kryo.register(classOf[org.ro= aringbitmap.ArrayContainer])
=C2=A0 =C2=A0 kryo.register(classOf[= org.apache.spark.scheduler.HighlyCompressedMapStatus])
=C2=A0 =C2= =A0 kryo.register(classOf[org.roaringbitmap.RoaringArray$Element])
=C2=A0 =C2=A0 kryo.register(classOf[org.roaringbitmap.RoaringArray$Elemen= t[]])
=C2=A0 =C2=A0 kryo.register(classOf[short[]])


in build file:

libraryDependencies +=3D "org.roaringbitmap"= ; % "RoaringBitmap" % "0.4.8"


This fails to compile:

...:53: identifier expected but ']' found.

[error] =C2= =A0 =C2=A0 kryo.register(classOf[org.roaringbitmap.RoaringArray$Element[]])=

also:

:54: identifier expected but ']' found.

[e= rror] =C2=A0 =C2=A0 kryo.register(classOf[short[]])

also:

:51: class HighlyCompressedMapStatus in package = scheduler cannot be accessed in package org.apache.spark.scheduler
[error] =C2=A0 =C2=A0 kryo.register(classOf[org.apache.spark.scheduler.Hi= ghlyCompressedMapStatus])


Sug= gestions?

Arun
=







--047d7bea40cc98003f05111bf99d--