spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Davidson <ilike...@gmail.com>
Subject Re: ADD_JARS doesn't properly work for spark-shell
Date Sun, 05 Jan 2014 02:28:09 GMT
Additionally, which version of Spark are you running?


On Sat, Jan 4, 2014 at 6:27 PM, Aaron Davidson <ilikerps@gmail.com> wrote:

> I am not an expert on these classpath issues, but if you're using local
> mode, you might also try to set SPARK_CLASSPATH to include the path to the
> jar file as well. This should not really help, since "adding jars" is the
> right way to get the jars to your executors (which is where the exception
> appears to be happening), but it would sure be interesting if it did.
>
>
> On Sat, Jan 4, 2014 at 4:50 PM, Aureliano Buendia <buendia360@gmail.com>wrote:
>
>> I should add that I can see in the log that the jar being shipped to the
>> workers:
>>
>> 14/01/04 15:34:52 INFO Executor: Fetching
>> http://192.168.1.111:51031/jars/my.jar.jar with timestamp 1388881979092
>> 14/01/04 15:34:52 INFO Utils: Fetching
>> http://192.168.1.111:51031/jars/my.jar.jar to
>> /var/folders/3g/jyx81ctj3698wbvphxhm4dw40000gn/T/fetchFileTemp8322008964976744710.tmp
>> 14/01/04 15:34:53 INFO Executor: Adding
>> file:/var/folders/3g/jyx81ctj3698wbvphxhm4dw40000gn/T/spark-d8ac8f66-fad6-4b3f-8059-73f13b86b070/my.jar.jar
>> to class loader
>>
>>
>> On Sun, Jan 5, 2014 at 12:46 AM, Aureliano Buendia <buendia360@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I'm trying to access my stand alone spark app from spark-shell. I tried
>>> starting the shell by:
>>>
>>> MASTER=local[2] ADD_JARS=/path/to/my/jar ./spark-shell
>>>
>>> The log shows that the jar file was loaded. Also, I can access and
>>> create a new instance of mypackage.MyClass.
>>>
>>> The problem is that myRDD.collect() returns RDD[MyClass], and that
>>> throws this exception:
>>>
>>> java.lang.ClassNotFoundException: mypackage.MyClass
>>>   at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>>   at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>   at java.security.AccessController.doPrivileged(Native Method)
>>>   at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>   at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>>>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>   at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>>>   at java.lang.Class.forName0(Native Method)
>>>   at java.lang.Class.forName(Class.java:264)
>>>   at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:622)
>>>   at
>>> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1593)
>>>   at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
>>>   at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1642)
>>>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341)
>>>   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
>>>   at org.apache.spark.util.Utils$.deserialize(Utils.scala:59)
>>>   at
>>> org.apache.spark.SparkContext$$anonfun$objectFile$1.apply(SparkContext.scala:573)
>>>   at
>>> org.apache.spark.SparkContext$$anonfun$objectFile$1.apply(SparkContext.scala:573)
>>>   at scala.collection.Iterator$$anon$21.hasNext(Iterator.scala:440)
>>>   at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:702)
>>>   at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:698)
>>>   at
>>> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:872)
>>>   at
>>> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:872)
>>>   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:107)
>>>   at org.apache.spark.scheduler.Task.run(Task.scala:53)
>>>   at
>>> org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:215)
>>>   at
>>> org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:50)
>>>   at
>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:182)
>>>   at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>   at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>   at java.lang.Thread.run(Thread.java:722)
>>>
>>> Does this mean that my jar was not shipped to the workers? Is this a
>>> known issue, or am I doing something wrong here?
>>>
>>
>>
>

Mime
View raw message