Hello Sachinpls paste the NoClassDefFound Exception so we can see what's failing, aslo please advise how are you running your Spark AppFor an extremely simple case, let's assume you have your MyFirstSparkApp packaged in your myFirstSparkApp.jarThen all you need to do would be to kick offspark-submit --class MyFirstSparkApp myFirstSparkApp.jarif you have any external dependencies (not spark , let's assume you are using common-utils.jar) then you should be able to kick it off viaspark-submit --class MyFirstSparkApp --jars common-utiils.jar myFirstSparkApp.jarI paste below the build.sbt i am using for my SparkExamples apps, hope this helps.krmarco
name := "SparkExamples"
version := "1.0"
scalaVersion := "2.10.5"
// Add a single dependency
libraryDependencies += "junit" % "junit" % "4.8" % "test"
libraryDependencies += "org.mockito" % "mockito-core" % "1.9.5"
libraryDependencies ++= Seq("org.slf4j" % "slf4j-api" % "1.7.5",
"org.slf4j" % "slf4j-simple" % "1.7.5",
"org.clapper" %% "grizzled-slf4j" % "1.0.2")
libraryDependencies += "org.powermock" % "powermock-mockito-release-full" % "1.5.4" % "test"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.6.1" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "1.6.1" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-streaming-flume" % "1.3.0" % "provided"
resolvers += "softprops-maven" at "http://dl.bintray.com/content/softprops/maven"On Wed, Jul 20, 2016 at 3:39 PM, Mich Talebzadeh <email@example.com> wrote:you need an uber jar file.Have you actually followed the dependencies and project sub-directory build?check this.http://stackoverflow.com/questions/28459333/how-to-build-an-uber-jar-fat-jar-using-sbt-within-intellij-ideaunder three answers the top one.I started reading the official SBT tutorial. .....HTH
Dr Mich Talebzadeh
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On 20 July 2016 at 09:54, Sachin Mittal <firstname.lastname@example.org> wrote:Here is my build.sbtSo as expected when I run the application via spark-submit I get the NoClassDefFoundError.Unlike in java application we build a standalone jar, which contains all the dependencies inside that jar, here all the dependencies are missing.So compile seems to be working fine. However when I inspect that jar, it only contains my scala class.When I run sbt package I see the jar created underI added all my dependencies via build.sbt (one dependency is under lib folder).For standalone scala application.Hi,I am following the example under https://spark.apache.org/docs/latest/quick-start.html
name := "Test Advice Project"
version := "1.0"
scalaVersion := "2.10.6"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.6.1",
"org.apache.spark" %% "spark-sql" % "1.6.1"
)Can anyone please guide me to as what is going wrong and why sbt package is not including all the dependencies jar classes in the new jar.ThanksSachinOn Tue, Jul 19, 2016 at 8:23 PM, Andrew Ehrlich <email@example.com> wrote:Yes, spark-core will depend on Hadoop and several other jars. Here’s the list of dependencies: https://github.com/apache/spark/blob/master/core/pom.xml#L35Whether you need spark-sql depends on whether you will use the DataFrame API. Without spark-sql, you will just have the RDD API.On Jul 19, 2016, at 7:09 AM, Sachin Mittal <firstname.lastname@example.org> wrote:Hi,Can someone please guide me what all jars I need to place in my lib folder of the project to build a standalone scala application via sbt.Note I need to provide static dependencies and I cannot download the jars using libraryDependencies.So I need to provide all the jars upfront.So far I found that we need:
Do we also need
hadoop-core-<version>.jarIs there any jar from spark side I may be missing? What I found that spark-core needs hadoop-core classes and if I don't add them then sbt was giving me this error:
[error] bad symbolic reference. A signature in SparkContext.class refers to term hadoop
[error] in package org.apache which is not available.So I was just confused on library dependency part when building an application via sbt. Any inputs here would be helpful.ThanksSachin