spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Iulian Dragoș <iulian.dra...@typesafe.com>
Subject Re: Error in SparkSQL/Scala IDE
Date Thu, 07 May 2015 08:24:05 GMT
On Thu, May 7, 2015 at 10:18 AM, Iulian Dragoș <iulian.dragos@typesafe.com>
wrote:

> Got it!
>
> I'll open a Jira ticket and PR when I have a working solution.
>

Scratch that, I found SPARK-5281
<https://issues.apache.org/jira/browse/SPARK-5281>..


>
> On Wed, May 6, 2015 at 11:53 PM, Michael Armbrust <michael@databricks.com>
> wrote:
>
>> Hi Iulian,
>>
>> The relevant code is in ScalaReflection
>> <https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala>,
>> and it would be awesome if you could suggest how to fix this more
>> generally. Specifically, this code is also broken when running from SBT:
>>
>>
>> $ build/sbt hive/console
>> scala> import implicits._
>> import implicits._
>>
>> scala> Seq((1,2)).toDF("a", "b")
>> scala.reflect.internal.MissingRequirementError: class
>> org.apache.spark.sql.catalyst.ScalaReflection in JavaMirror with
>> java.net.URLClassLoader@752d2e33 of type class java.net.URLClassLoader
>> with classpath
>> [file:/root/.sbt/boot/scala-2.10.4/lib/jline.jar,file:/root/.sbt/boot/scala-2.10.4/lib/scala-library.jar,file:/root/.sbt/boot/scala-2.10.4/lib/scala-compiler.jar,file:/root/.sbt/boot/scala-2.10.4/lib/jansi.jar,file:/root/.sbt/boot/scala-2.10.4/lib/scala-reflect.jar]
>> and parent being xsbt.boot.BootFilteredLoader@625e4591 of type class
>> xsbt.boot.BootFilteredLoader with classpath [<unknown>] and parent being
>> sun.misc.Launcher$AppClassLoader@1a8c064 of type class
>> sun.misc.Launcher$AppClassLoader with classpath
>> [file:/root/spark/build/sbt-launch-0.13.7.jar] and parent being
>> sun.misc.Launcher$ExtClassLoader@6deee615 of type class
>> sun.misc.Launcher$ExtClassLoader with classpath
>> [file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunjce_provider.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/
>> nsns.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/java-atk-wrapper.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunpkcs11.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/zipfs.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/localedata.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/icedtea-sound.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/libatk-wrapper.so]
>> and parent being primordial classloader with boot classpath
>> [/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rhino.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/classes]
>> not found.
>>
>> Michael
>>
>> On Wed, May 6, 2015 at 2:12 AM, Iulian Dragoș <iulian.dragos@typesafe.com
>> > wrote:
>>
>>> Hi, I just saw this question. I posted my solution to this stack
>>> overflow question.
>>> <https://stackoverflow.com/questions/29796928/whats-the-most-efficient-way-to-filter-a-dataframe>
>>>
>>> Scala reflection can take a classloader when creating a mirror (
>>> universe.runtimeMirror(loader)). I can have a look, but I didn’t find
>>> much about mirrors in spark-sql.
>>>
>>> iulian
>>> ​
>>>
>>
>>
>
>
> --
>
> --
> Iulian Dragos
>
> ------
> Reactive Apps on the JVM
> www.typesafe.com
>
>


-- 

--
Iulian Dragos

------
Reactive Apps on the JVM
www.typesafe.com

Mime
View raw message