spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Programmatically running spark jobs using yarn-client
Date Mon, 08 Dec 2014 16:09:42 GMT
How are you submitting the job? You need to create a jar of your code (sbt
package will give you one inside target/scala-*/projectname-*.jar) and then
use it while submitting. If you are not using spark-submit then you can
simply add this jar to spark by
sc.addJar("/path/to/target/scala*/projectname*jar")

Thanks
Best Regards

On Mon, Dec 8, 2014 at 7:23 PM, Aniket Bhatnagar <aniket.bhatnagar@gmail.com
> wrote:

> I am trying to create (yet another) spark as a service tool that lets you
> submit jobs via REST APIs. I think I have nearly gotten it to work baring a
> few issues. Some of which seem already fixed in 1.2.0 (like SPARK-2889) but
> I have hit the road block with the following issue.
>
> I have created a simple spark job as following:
>
> class StaticJob {
> import SparkContext._
> override def run(sc: SparkContext): Result = {
>   val array = Range(1, 10000000).toArray
>   val rdd = sc.parallelize(array)
>   val paired = rdd.map(i => (i % 10000, i)).sortByKey()
>   val sum = paired.countByKey()
>   SimpleResult(sum)
> }
> }
>
> When I submit this job programmatically, it gives me a class not found
> error:
>
> 2014-12-08 05:41:18,421 [Result resolver thread-0] [warn]
> o.a.s.s.TaskSetManager - Lost task 0.0 in stage 0.0 (TID 0,
> localhost.localdomain): java.lang.ClassNotFoundException:
> com.blah.server.examples.StaticJob$$anonfun$1
>         java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>         java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>         java.security.AccessController.doPrivileged(Native Method)
>         java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>         java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>         java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>         java.lang.Class.forName0(Native Method)
>         java.lang.Class.forName(Class.java:270)
>
> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59)
>
> I decompiled the StaticJob$$anonfun$1 class and it seems to point to
> closure 'rdd.map(i => (i % 10000, i))'. I am sure why this is happening.
> Any help will be greatly appreciated.
>

Mime
View raw message