spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrei <faithlessfri...@gmail.com>
Subject Is uberjar a recommended way of running Spark/Scala applications?
Date Thu, 29 May 2014 12:24:02 GMT
I'm using Spark 1.0 and sbt assembly plugin to create uberjar of my
application. However, when I run assembly command, I get a number of errors
like this:

java.lang.RuntimeException: deduplicate: different file contents found in
the following:
/home/username/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log$Logger.class
/home/username/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log$Logger.class
...

As far as I can see, Spark Core depends on both - Minlog and Kryo, and the
latter includes Minlog classes itself. Classes are binary different, so
assembly can't combine them. And there's a number of such conflicts - I
fixed some of them manually via mergeStrategy, but list of exceptions
becomes larger and larger. I can continues, but it just does't look like
the right way.

My questions are:

1. Is an uberjar a recommended way of running Spark applications?
2. If so, should I include Spark itself into this large jar?
3. If not, what is a recommended way to do both - development and
deployment (assuming ordinary sbt project).

Thanks,
Andrei

Mime
View raw message