spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramkumar Venkataraman <ram.the.m...@gmail.com>
Subject [Spark Streaming] NoClassDefFoundError : StateSpec
Date Thu, 12 Jan 2017 15:35:10 GMT
Spark: 1.6.1

I am trying to use the new mapWithState API and I am getting the following
error:

Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/spark/streaming/StateSpec$
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.streaming.StateSpec$

Build.sbt
====================
scalaVersion := "2.10.6"
typelevelDefaultSettings
val sparkVersion = "1.6.1"

resolvers ++= Seq(
  "Sonatype OSS Snapshots" at
"https://oss.sonatype.org/content/repositories/snapshots"
)

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-streaming-kafka" % sparkVersion,
  "com.fasterxml.jackson.core" % "jackson-databind" % "2.3.3" // Needed by
spark-core
)
==========

This is how my spark-submit looks like:

./bin/spark-submit --verbose --master yarn-client  --num-executors 50
--driver-memory=4G --executor-memory=8G   --conf
"spark.driver.extraJavaOptions=-XX:MaxPermSize=6G -XX:+UseConcMarkSweepGC"
--conf "spark.executor.extraJavaOptions=-XX:+UseConcMarkSweepGC -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps"  --class MY_DRIVER
~/project-assembly-0.0.1-SNAPSHOT.jar

==========

Is there anything I am missing here? I understand that NoClassDefFoundError
means the required Jars aren't present in the classpath, I am just not able
to understand why this class alone is missing, when the others related to
window, etc. are found. Do I have to pass in additional jars to make this
API work?




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-NoClassDefFoundError-StateSpec-tp28301.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message