spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Dong <>
Subject How to specify PATHS for user defined functions.
Date Thu, 09 Jul 2015 22:07:34 GMT
Hi, All,
  I have a function and want to access it in my spark programs, but I got
"Exception in thread "main" java.lang.NoSuchMethodError" in spark-submit. I
put the function under:


object MYFUNC{
  def FUNC1(input: List[String]) = {

and in my Spark program I import it like:

   val aaa=List("import", "org", "apache", "spark", "SparkContext")
   val res=MYFUNC.FUNC1(aaa)

But after I "sbt package" and set the CLASSPATH and spark-submit the
program I got the above error. It's strange that I can import this package
and run the function of "val res=MYFUNC.FUNC1(aaa)" under a spark-shell
successfully. What's the possible problems? Thanks!


View raw message