spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Ehrlich <and...@aehrlich.com>
Subject Re: Building standalone spark application via sbt
Date Tue, 19 Jul 2016 14:53:16 GMT
Yes, spark-core will depend on Hadoop and several other jars.  Here’s the list of dependencies:
https://github.com/apache/spark/blob/master/core/pom.xml#L35 <https://github.com/apache/spark/blob/master/core/pom.xml#L35>

Whether you need spark-sql depends on whether you will use the DataFrame API. Without spark-sql,
you will just have the RDD API.

> On Jul 19, 2016, at 7:09 AM, Sachin Mittal <sjmittal@gmail.com> wrote:
> 
> 
> Hi,
> Can someone please guide me what all jars I need to place in my lib folder of the project
to build a standalone scala application via sbt.
> 
> Note I need to provide static dependencies and I cannot download the jars using libraryDependencies.
> So I need to provide all the jars upfront.
> 
> So far I found that we need:
> spark-core_<version>.jar
> 
> Do we also need
> spark-sql_<version>.jar
> and
> hadoop-core-<version>.jar
> 
> Is there any jar from spark side I may be missing? What I found that spark-core needs
hadoop-core classes and if I don't add them then sbt was giving me this error:
> [error] bad symbolic reference. A signature in SparkContext.class refers to term hadoop
> [error] in package org.apache which is not available.
> 
> So I was just confused on library dependency part when building an application via sbt.
Any inputs here would be helpful.
> 
> Thanks
> Sachin
> 
> 
> 


Mime
View raw message