spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: When using spark shell, classpath on workers does not seem to see all of my custom classes
Date Sun, 08 Sep 2013 23:01:56 GMT
Which version of Spark is this with? Did the logs print something about sending the JAR you
added with ADD_JARS to the cluster?

Matei

On Sep 8, 2013, at 8:56 AM, Gary Malouf <malouf.gary@gmail.com> wrote:

> I built a custom jar with among other things, nscalatime and joda time packed inside
of it.  Using the ADD_JARS variable, I have added this super jar to my classpath on the scheduler
when running spark-shell.  I wrote a function that grabs protobuf data, filters and then maps
each message to a (LocalDate, Option[String]) format.  Unfortunately, this does not run and
I get the following:
> 
> 13/09/08 15:50:43 INFO cluster.TaskSetManager: Finished TID 6 in 348 ms (progress: 7/576)
> Exception in thread "Thread-159" java.lang.ClassNotFoundException: org.joda.time.LocalDate
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>     at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.scala$tools$nsc$util$ScalaClassLoader$$super$findClass(ScalaClassLoader.scala:88)
>     at scala.tools.nsc.util.ScalaClassLoader$class.findClass(ScalaClassLoader.scala:44)
>     at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.findClass(ScalaClassLoader.scala:88)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>     at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.scala$tools$nsc$util$ScalaClassLoader$$super$loadClass(ScalaClassLoader.scala:88)
>     at scala.tools.nsc.util.ScalaClassLoader$class.loadClass(ScalaClassLoader.scala:50)
>     at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.loadClass(ScalaClassLoader.scala:88)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>     at java.lang.Class.forName0(Native Method)
>     at java.lang.Class.forName(Class.java:266)
>     at spark.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:20)
>     at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1610)
>     at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
>     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769)
>     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
>     at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>     at it.unimi.dsi.fastutil.objects.Object2LongOpenHashMap.readObject(Object2LongOpenHashMap.java:757)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:601)
>     at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
>     at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1891)
>     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
>     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
>     at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
>     at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913)
>     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
>     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
>     at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>     at spark.scheduler.TaskResult.readExternal(TaskResult.scala:26)
>     at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1835)
>     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1794)
>     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
>     at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>     at spark.JavaDeserializationStream.readObject(JavaSerializer.scala:23)
>     at spark.JavaSerializerInstance.deserialize(JavaSerializer.scala:45)
>     at spark.scheduler.cluster.TaskSetManager.taskFinished(TaskSetManager.scala:261)
>     at spark.scheduler.cluster.TaskSetManager.statusUpdate(TaskSetManager.scala:236)
>     at spark.scheduler.cluster.ClusterScheduler.statusUpdate(ClusterScheduler.scala:219)
>     at spark.scheduler.mesos.MesosSchedulerBackend.statusUpdate(MesosSchedulerBackend.scala:264)
> 13/09/08 15:50:43 INFO mesos.MesosSchedulerBackend: driver.run() returned with code DRIVER_ABORTED
> 
> 
> The code definitely compiles in the interpreter and the executors seem to find the protobuf
messages which are in the same jar - any idea what could be causing the problem?  


Mime
View raw message