spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Dong <dongda...@gmail.com>
Subject How to specify PATHS for user defined functions.
Date Thu, 09 Jul 2015 22:07:34 GMT
Hi, All,
  I have a function and want to access it in my spark programs, but I got
the:
"Exception in thread "main" java.lang.NoSuchMethodError" in spark-submit. I
put the function under:
./src/main/scala/com/aaa/MYFUNC/MYFUNC.scala:

package com.aaa.MYFUNC

object MYFUNC{
  def FUNC1(input: List[String]) = {
  ......
    }
  }



and in my Spark program I import it like:

import com.aaa.MYFUNC._
...
   val aaa=List("import", "org", "apache", "spark", "SparkContext")
   val res=MYFUNC.FUNC1(aaa)
...

But after I "sbt package" and set the CLASSPATH and spark-submit the
program I got the above error. It's strange that I can import this package
and run the function of "val res=MYFUNC.FUNC1(aaa)" under a spark-shell
successfully. What's the possible problems? Thanks!

Cheers,
Dan

Mime
View raw message