Thanks Akhil. I was wondering why it isn&#39;t available to find the class even though it existed in the same class loader as SparkContext. As a workaround, I used the following code the add all dependent jars in a playframework application to spark context.<div><br></div><div><div>@tailrec</div><div>private def addClassPathJars(sparkContext: SparkContext, classLoader: ClassLoader): Unit = {</div><div>  classLoader match {</div><div>    case urlClassLoader: URLClassLoader =&gt; {</div><div>      urlClassLoader.getURLs.foreach(classPathUrl =&gt; {</div><div>        if (classPathUrl.toExternalForm.endsWith(&quot;.jar&quot;)) {</div><div>          LOGGER.debug(s&quot;Added $classPathUrl to spark context $sparkContext&quot;)</div><div>          sparkContext.addJar(classPathUrl.toExternalForm)</div><div>        } else {</div><div>          LOGGER.<span style="line-height:19.7999992370605px">debug</span>(s&quot;Ignored $classPathUrl while adding to spark context $sparkContext&quot;)</div><div>        }</div><div>      })</div><div>    }</div><div>    case _ =&gt; LOGGER.<span style="line-height:19.7999992370605px">debug</span>(s&quot;Ignored class loader $classLoader as it does not subclasses URLClassLoader&quot;)</div><div>  }</div><div>  if (classLoader.getParent != null){</div><div>    addClassPathJars(sparkContext, classLoader.getParent)</div><div>  }</div><div>}</div></div><br><div class="gmail_quote">On Mon Dec 08 2014 at 21:39:42 Akhil Das &lt;<a href="mailto:akhil@sigmoidanalytics.com">akhil@sigmoidanalytics.com</a>&gt; wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:courier new,monospace;color:#000000">How are you submitting the job? You need to create a jar of your code (sbt package will give you one inside target/scala-*/projectname-*.jar) and then use it while submitting. If you are not using spark-submit then you can simply add this jar to spark by sc.addJar(&quot;/path/to/target/scala*/projectname*jar&quot;)</div></div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr">Thanks<div>Best Regards</div></div></div></div></div><div class="gmail_extra">
<br><div class="gmail_quote">On Mon, Dec 8, 2014 at 7:23 PM, Aniket Bhatnagar <span dir="ltr">&lt;<a href="mailto:aniket.bhatnagar@gmail.com" target="_blank">aniket.bhatnagar@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I am trying to create (yet another) spark as a service tool that lets you submit jobs via REST APIs. I think I have nearly gotten it to work baring a few issues. Some of which seem already fixed in 1.2.0 (like SPARK-2889) but I have hit the road block with the following issue. <div><br></div><div>I have created a simple spark job as following:</div><div><br></div><div>class StaticJob {</div><div>import SparkContext._<br></div><div>override def run(sc: SparkContext): Result = {</div><div><div>  val array = Range(1, 10000000).toArray</div><div>  val rdd = sc.parallelize(array)</div><div>  val paired = rdd.map(i =&gt; (i % 10000, i)).sortByKey()</div><div>  val sum = paired.countByKey()</div><div>  SimpleResult(sum)</div><div>}</div></div><div>}</div><div><br></div><div>When I submit this job programmatically, it gives me a class not found error: </div><div><br></div><div><div>2014-12-08 05:41:18,421 [Result resolver thread-0] [warn] o.a.s.s.TaskSetManager - Lost task 0.0 in stage 0.0 (TID 0, localhost.localdomain): java.lang.ClassNotFoundException: com.blah.server.examples.StaticJob$$anonfun$1</div><div>        java.net.URLClassLoader$1.run(URLClassLoader.java:366)</div><div>        java.net.URLClassLoader$1.run(URLClassLoader.java:355)</div><div>        java.security.AccessController.doPrivileged(Native Method)</div><div>        java.net.URLClassLoader.findClass(URLClassLoader.java:354)</div><div>        java.lang.ClassLoader.loadClass(ClassLoader.java:425)</div><div>        java.lang.ClassLoader.loadClass(ClassLoader.java:358)</div><div>        java.lang.Class.forName0(Native Method)</div><div>        java.lang.Class.forName(Class.java:270)</div><div>        org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59)</div></div><div><br></div><div>I decompiled the StaticJob$$anonfun$1 class and it seems to point to closure &#39;rdd.map(i =&gt; (i % 10000, i))&#39;. I am sure why this is happening. Any help will be greatly appreciated.</div>
</blockquote></div><br></div></blockquote></div>