spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramkumar Chokkalingam <ramkumar...@gmail.com>
Subject Re: Dependency while creating jar duplicate file.
Date Sun, 27 Oct 2013 17:04:34 GMT
Hey Patrick,

Thanks for the mail. So, I did solve the issue by *using the MergeStrategy** .
*I was more concerned because Spark was only major dependency - Since it
was a prebuild version and failed, I wanted to check with you.


Regards,

Ramkumar Chokkalingam ,
University of Washington.
LinkedIn <http://www.linkedin.com/in/mynameisram>





On Sun, Oct 27, 2013 at 9:33 AM, Patrick Wendell <pwendell@gmail.com> wrote:

> Hey Ram,
>
> When you create the assembly Jar for your own project, you'll need to
> deal with all possible conflicts. And this includes various conflicts
> inside of Spark's dependencies.
>
> I've noticed the Apache commons libraries often have conflicts. One
> thing you could do is set MergeStrategy.first for all class fils with
> the path org/apache/common/*.
>
> You can also add a catch-all default strategy, like the one in the
> Spark build. In fact, it might make sense to just copy over the policy
> from the Spark build as a starting point:
>
>
> https://github.com/apache/incubator-spark/blob/master/project/SparkBuild.scala#L332
>
> - Patrick
>
>
> On Sun, Oct 27, 2013 at 1:04 AM, Ramkumar Chokkalingam
> <ramkumar.au@gmail.com> wrote:
> >
> > Hello Spark Community,
> >
> > I'm trying to convert my project into a single JAR. I used sbt assembly
> utility to do the same.
> >
> > This is the error I got,
> >
> > [error] (*:assembly) deduplicate: different file contents found in the
> following:
> > [error]
> /usr/local/spark/ram_examples/rm/lib_managed/jars/org.mortbay.jetty/servlet-api/servlet-api-2.5-20081211.jar:javax/servlet/SingleThreadModel.class
> > [error]
> /usr/local/spark/ram_examples/rm/lib_managed/orbits/org.eclipse.jetty.orbit/javax.servlet/javax.servlet-2.5.0.v201103041518.jar:javax/servlet/SingleThreadModel.class
> >
> > So when I fix that with,
> https://github.com/sbt/sbt-assembly#merge-strategy, i face another
> dependency,
> >
> > java.lang.RuntimeException: deduplicate: different file contents found
> in the following:
> >
> /usr/local/spark/ram_examples/rm/lib_managed/jars/commons-beanutils/commons-beanutils-core/commons-beanutils-core-1.8.0.jar:org/apache/commons/beanutils/converters/FloatArrayConverter.class
> >
> /usr/local/spark/ram_examples/rm/lib_managed/jars/commons-beanutils/commons-beanutils/commons-beanutils-1.7.0.jar:org/apache/commons/beanutils/converters/FloatArrayConverter.class
> >
> > This is my build file  http://pastebin.com/5W9f1g1e
> >
> > While this seems like an error that has been already discussed and
> solved, and discussed here . I get this in latest Spark [version 0.8.0].
> I'm just curious because, this is clean build of Spark that I'm using and
> it seems to work fine (with sbt run/sbt package),but when I use sbt
> assembly I get this error. Am I missing something while creating JAR ? Any
> help would be appreciated. Thanks!
> >
> >
> > Regards,
> > R
> > am
> >
> >
>

Mime
View raw message