spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jakob Odersky <ja...@odersky.com>
Subject Re: Error building spark app with Maven
Date Tue, 15 Mar 2016 22:18:15 GMT
Hi Mich,
probably unrelated to the current error you're seeing, however the
following dependencies will bite you later:
<artifactId>spark-hive_2.10</artifactId>
<artifactId>spark-csv_2.11</artifactId>
the problem here is that you're using libraries built for different
Scala binary versions (the numbers after the underscore). The simple
fix here is to specify the Scala binary version you're project builds
for (2.10 in your case, however note that version is EOL, you should
upgrade to scala 2.11.8 if possible).

On a side note, sbt takes care of handling correct scala versions for
you (the double %% actually is a shorthand for appending
"_scalaBinaryVersion" to your dependency). It also enables you to
build and publish your project seamlessly against multiple versions. I
would strongly recommend to use it in Scala projects.

cheers,
--Jakob



On Tue, Mar 15, 2016 at 3:08 PM, Mich Talebzadeh
<mich.talebzadeh@gmail.com> wrote:
> Hi,
>
> I normally use sbt and using this sbt file works fine for me
>
>  cat ImportCSV.sbt
> name := "ImportCSV"
> version := "1.0"
> scalaVersion := "2.10.4"
> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1"
> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1"
> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1"
> libraryDependencies += "com.databricks" % "spark-csv_2.11" % "1.3.0"
>
> This is my first trial using Mavan and pom
>
>
> my pom.xml file looks like this but throws error at build
>
>
> [DEBUG]       com.univocity:univocity-parsers:jar:1.5.1:compile
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 1.326 s
> [INFO] Finished at: 2016-03-15T22:17:29+00:00
> [INFO] Final Memory: 14M/455M
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] Failed to execute goal on project scala: Could not resolve
> dependencies for project spark:scala:jar:1.0: The following artifacts could
> not be resolved: org.apache.spark:spark-core:jar:1.5.1,
> org.apache.spark:spark-sql:jar:1.5.1: Failure to find
> org.apache.spark:spark-core:jar:1.5.1 in
> https://repo.maven.apache.org/maven2 was cached in the local repository,
> resolution will not be reattempted until the update interval of central has
> elapsed or updates are forced -> [Help 1]
>
>
> My pom file is
>
>
>  cat pom.xml
> <project xmlns="http://maven.apache.org/POM/4.0.0"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/maven-v4_0_0.xsd">
> <modelVersion>4.0.0</modelVersion>
> <groupId>spark</groupId>
> <version>1.0</version>
> <name>${project.artifactId}</name>
>
> <properties>
> <maven.compiler.source>1.7</maven.compiler.source>
> <maven.compiler.target>1.7</maven.compiler.target>
> <encoding>UTF-8</encoding>
> <scala.version>2.10.4</scala.version>
> <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
> </properties>
>
> <dependencies>
>   <dependency>
>     <groupId>org.scala-lang</groupId>
>     <artifactId>scala-library</artifactId>
>     <version>2.10.2</version>
>   </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-core</artifactId>
> <version>1.5.1</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-sql</artifactId>
> <version>1.5.1</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-hive_2.10</artifactId>
> <version>1.5.0</version>
> </dependency>
> <dependency>
> <groupId>com.databricks</groupId>
> <artifactId>spark-csv_2.11</artifactId>
> <version>1.3.0</version>
> </dependency>
> </dependencies>
>
> <build>
> <sourceDirectory>src/main/scala</sourceDirectory>
> <plugins>
> <plugin>
> <groupId>org.scala-tools</groupId>
> <artifactId>maven-scala-plugin</artifactId>
> <version>${maven-scala-plugin.version}</version>
> <executions>
> <execution>
> <goals>
> <goal>compile</goal>
> </goals>
> </execution>
> </executions>
> <configuration>
> <jvmArgs>
> <jvmArg>-Xms64m</jvmArg>
> <jvmArg>-Xmx1024m</jvmArg>
> </jvmArgs>
> </configuration>
> </plugin>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-shade-plugin</artifactId>
> <version>1.6</version>
> <executions>
> <execution>
> <phase>package</phase>
> <goals>
> <goal>shade</goal>
> </goals>
> <configuration>
> <filters>
> <filter>
> <artifact>*:*</artifact>
> <excludes>
> <exclude>META-INF/*.SF</exclude>
> <exclude>META-INF/*.DSA</exclude>
> <exclude>META-INF/*.RSA</exclude>
> </excludes>
> </filter>
> </filters>
> <transformers>
> <transformer
> implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
> <mainClass>com.group.id.Launcher1</mainClass>
> </transformer>
> </transformers>
> </configuration>
> </execution>
> </executions>
> </plugin>
> </plugins>
> </build>
>
> <artifactId>scala</artifactId>
> </project>
>
>
> I am sure I have omitted something?
>
>
> Thanks
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
>
>
> http://talebzadehmich.wordpress.com
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message