spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jakob Odersky <ja...@odersky.com>
Subject Re: Error building spark app with Maven
Date Tue, 15 Mar 2016 23:34:07 GMT
The artifactId in maven basically (in a simple case) corresponds to name in sbt.

Note however that you will manually need to append the
_scalaBinaryVersion to the artifactId in case you would like to build
against multiple scala versions (otherwise maven will overwrite the
generated jar with the latest one).


On Tue, Mar 15, 2016 at 4:27 PM, Mich Talebzadeh
<mich.talebzadeh@gmail.com> wrote:
> ok  Ted
>
> In sbt I have
>
> name := "ImportCSV"
> version := "1.0"
> scalaVersion := "2.10.4"
>
> which ends up in importcsv_2.10-1.0.jar as part of
> target/scala-2.10/importcsv_2.10-1.0.jar
>
> In mvn I have
>
> <version>1.0</version>
> <artifactId>scala</artifactId>
>
>
> Does it matter?
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
>
> On 15 March 2016 at 23:17, Ted Yu <yuzhihong@gmail.com> wrote:
>>
>> <version>1.0</version>
>> ...
>> <artifactId>scala</artifactId>
>>
>> On Tue, Mar 15, 2016 at 4:14 PM, Mich Talebzadeh
>> <mich.talebzadeh@gmail.com> wrote:
>>>
>>> An observation
>>>
>>> Once compiled with MVN the job submit works as follows:
>>>
>>> + /usr/lib/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --packages
>>> com.databricks:spark-csv_2.11:1.3.0 --class ImportCSV --master
>>> spark://50.140.197.217:7077 --executor-memory=12G --executor-cores=12
>>> --num-executors=2 target/scala-1.0.jar
>>>
>>> With sbt it takes this form
>>>
>>> + /usr/lib/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --packages
>>> com.databricks:spark-csv_2.11:1.3.0 --class ImportCSV --master
>>> spark://50.140.197.217:7077 --executor-memory=12G --executor-cores=12
>>> --num-executors=2 target/scala-2.10/importcsv_2.10-1.0.jar
>>>
>>> They both return the same results. However, why mvnjar file name is
>>> different (may be a naive question!)?
>>>
>>> thanks
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn
>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>>
>>> On 15 March 2016 at 22:43, Mich Talebzadeh <mich.talebzadeh@gmail.com>
>>> wrote:
>>>>
>>>> Many thanks Ted and thanks for heads up Jakob
>>>>
>>>> Just these two changes to dependencies
>>>>
>>>> <dependency>
>>>> <groupId>org.apache.spark</groupId>
>>>> <artifactId>spark-core_2.10</artifactId>
>>>> <version>1.5.1</version>
>>>> </dependency>
>>>> <dependency>
>>>> <groupId>org.apache.spark</groupId>
>>>> <artifactId>spark-sql_2.10</artifactId>
>>>> <version>1.5.1</version>
>>>> </dependency>
>>>>
>>>>
>>>> [DEBUG] endProcessChildren: artifact=spark:scala:jar:1.0
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] BUILD SUCCESS
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] Total time: 01:04 min
>>>> [INFO] Finished at: 2016-03-15T22:55:08+00:00
>>>> [INFO] Final Memory: 32M/1089M
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>>
>>>> LinkedIn
>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>>
>>>>
>>>> On 15 March 2016 at 22:18, Jakob Odersky <jakob@odersky.com> wrote:
>>>>>
>>>>> Hi Mich,
>>>>> probably unrelated to the current error you're seeing, however the
>>>>> following dependencies will bite you later:
>>>>> <artifactId>spark-hive_2.10</artifactId>
>>>>> <artifactId>spark-csv_2.11</artifactId>
>>>>> the problem here is that you're using libraries built for different
>>>>> Scala binary versions (the numbers after the underscore). The simple
>>>>> fix here is to specify the Scala binary version you're project builds
>>>>> for (2.10 in your case, however note that version is EOL, you should
>>>>> upgrade to scala 2.11.8 if possible).
>>>>>
>>>>> On a side note, sbt takes care of handling correct scala versions for
>>>>> you (the double %% actually is a shorthand for appending
>>>>> "_scalaBinaryVersion" to your dependency). It also enables you to
>>>>> build and publish your project seamlessly against multiple versions.
I
>>>>> would strongly recommend to use it in Scala projects.
>>>>>
>>>>> cheers,
>>>>> --Jakob
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Mar 15, 2016 at 3:08 PM, Mich Talebzadeh
>>>>> <mich.talebzadeh@gmail.com> wrote:
>>>>> > Hi,
>>>>> >
>>>>> > I normally use sbt and using this sbt file works fine for me
>>>>> >
>>>>> >  cat ImportCSV.sbt
>>>>> > name := "ImportCSV"
>>>>> > version := "1.0"
>>>>> > scalaVersion := "2.10.4"
>>>>> > libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1"
>>>>> > libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1"
>>>>> > libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1"
>>>>> > libraryDependencies += "com.databricks" % "spark-csv_2.11" % "1.3.0"
>>>>> >
>>>>> > This is my first trial using Mavan and pom
>>>>> >
>>>>> >
>>>>> > my pom.xml file looks like this but throws error at build
>>>>> >
>>>>> >
>>>>> > [DEBUG]       com.univocity:univocity-parsers:jar:1.5.1:compile
>>>>> > [INFO]
>>>>> >
>>>>> > ------------------------------------------------------------------------
>>>>> > [INFO] BUILD FAILURE
>>>>> > [INFO]
>>>>> >
>>>>> > ------------------------------------------------------------------------
>>>>> > [INFO] Total time: 1.326 s
>>>>> > [INFO] Finished at: 2016-03-15T22:17:29+00:00
>>>>> > [INFO] Final Memory: 14M/455M
>>>>> > [INFO]
>>>>> >
>>>>> > ------------------------------------------------------------------------
>>>>> > [ERROR] Failed to execute goal on project scala: Could not resolve
>>>>> > dependencies for project spark:scala:jar:1.0: The following artifacts
>>>>> > could
>>>>> > not be resolved: org.apache.spark:spark-core:jar:1.5.1,
>>>>> > org.apache.spark:spark-sql:jar:1.5.1: Failure to find
>>>>> > org.apache.spark:spark-core:jar:1.5.1 in
>>>>> > https://repo.maven.apache.org/maven2 was cached in the local
>>>>> > repository,
>>>>> > resolution will not be reattempted until the update interval of
>>>>> > central has
>>>>> > elapsed or updates are forced -> [Help 1]
>>>>> >
>>>>> >
>>>>> > My pom file is
>>>>> >
>>>>> >
>>>>> >  cat pom.xml
>>>>> > <project xmlns="http://maven.apache.org/POM/4.0.0"
>>>>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>> > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>>>>> > http://maven.apache.org/maven-v4_0_0.xsd">
>>>>> > <modelVersion>4.0.0</modelVersion>
>>>>> > <groupId>spark</groupId>
>>>>> > <version>1.0</version>
>>>>> > <name>${project.artifactId}</name>
>>>>> >
>>>>> > <properties>
>>>>> > <maven.compiler.source>1.7</maven.compiler.source>
>>>>> > <maven.compiler.target>1.7</maven.compiler.target>
>>>>> > <encoding>UTF-8</encoding>
>>>>> > <scala.version>2.10.4</scala.version>
>>>>> > <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
>>>>> > </properties>
>>>>> >
>>>>> > <dependencies>
>>>>> >   <dependency>
>>>>> >     <groupId>org.scala-lang</groupId>
>>>>> >     <artifactId>scala-library</artifactId>
>>>>> >     <version>2.10.2</version>
>>>>> >   </dependency>
>>>>> > <dependency>
>>>>> > <groupId>org.apache.spark</groupId>
>>>>> > <artifactId>spark-core</artifactId>
>>>>> > <version>1.5.1</version>
>>>>> > </dependency>
>>>>> > <dependency>
>>>>> > <groupId>org.apache.spark</groupId>
>>>>> > <artifactId>spark-sql</artifactId>
>>>>> > <version>1.5.1</version>
>>>>> > </dependency>
>>>>> > <dependency>
>>>>> > <groupId>org.apache.spark</groupId>
>>>>> > <artifactId>spark-hive_2.10</artifactId>
>>>>> > <version>1.5.0</version>
>>>>> > </dependency>
>>>>> > <dependency>
>>>>> > <groupId>com.databricks</groupId>
>>>>> > <artifactId>spark-csv_2.11</artifactId>
>>>>> > <version>1.3.0</version>
>>>>> > </dependency>
>>>>> > </dependencies>
>>>>> >
>>>>> > <build>
>>>>> > <sourceDirectory>src/main/scala</sourceDirectory>
>>>>> > <plugins>
>>>>> > <plugin>
>>>>> > <groupId>org.scala-tools</groupId>
>>>>> > <artifactId>maven-scala-plugin</artifactId>
>>>>> > <version>${maven-scala-plugin.version}</version>
>>>>> > <executions>
>>>>> > <execution>
>>>>> > <goals>
>>>>> > <goal>compile</goal>
>>>>> > </goals>
>>>>> > </execution>
>>>>> > </executions>
>>>>> > <configuration>
>>>>> > <jvmArgs>
>>>>> > <jvmArg>-Xms64m</jvmArg>
>>>>> > <jvmArg>-Xmx1024m</jvmArg>
>>>>> > </jvmArgs>
>>>>> > </configuration>
>>>>> > </plugin>
>>>>> > <plugin>
>>>>> > <groupId>org.apache.maven.plugins</groupId>
>>>>> > <artifactId>maven-shade-plugin</artifactId>
>>>>> > <version>1.6</version>
>>>>> > <executions>
>>>>> > <execution>
>>>>> > <phase>package</phase>
>>>>> > <goals>
>>>>> > <goal>shade</goal>
>>>>> > </goals>
>>>>> > <configuration>
>>>>> > <filters>
>>>>> > <filter>
>>>>> > <artifact>*:*</artifact>
>>>>> > <excludes>
>>>>> > <exclude>META-INF/*.SF</exclude>
>>>>> > <exclude>META-INF/*.DSA</exclude>
>>>>> > <exclude>META-INF/*.RSA</exclude>
>>>>> > </excludes>
>>>>> > </filter>
>>>>> > </filters>
>>>>> > <transformers>
>>>>> > <transformer
>>>>> >
>>>>> > implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
>>>>> > <mainClass>com.group.id.Launcher1</mainClass>
>>>>> > </transformer>
>>>>> > </transformers>
>>>>> > </configuration>
>>>>> > </execution>
>>>>> > </executions>
>>>>> > </plugin>
>>>>> > </plugins>
>>>>> > </build>
>>>>> >
>>>>> > <artifactId>scala</artifactId>
>>>>> > </project>
>>>>> >
>>>>> >
>>>>> > I am sure I have omitted something?
>>>>> >
>>>>> >
>>>>> > Thanks
>>>>> >
>>>>> >
>>>>> > Dr Mich Talebzadeh
>>>>> >
>>>>> >
>>>>> >
>>>>> > LinkedIn
>>>>> >
>>>>> > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>> >
>>>>> >
>>>>> >
>>>>> > http://talebzadehmich.wordpress.com
>>>>> >
>>>>> >
>>>>
>>>>
>>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message