spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Boesch <java...@gmail.com>
Subject Re: Spark: issues with running a sbt fat jar due to akka dependencies
Date Fri, 02 May 2014 04:31:55 GMT
Hi Shivani,
    Your work would be helpful to others (well at least to me ;) Would you
be willing to share your resultant sbt build files?



2014-05-01 17:45 GMT-07:00 Shivani Rao <raoshivani@gmail.com>:

> Hello Koert,
>
> That did not work. I specified it in my email already. But I figured a way
> around it  by excluding akka dependencies
>
> Shivani
>
>
> On Tue, Apr 29, 2014 at 12:37 PM, Koert Kuipers <koert@tresata.com> wrote:
>
>> you need to merge reference.conf files and its no longer an issue.
>>
>> see the Build for for spark itself:
>>   case "reference.conf" => MergeStrategy.concat
>>
>>
>> On Tue, Apr 29, 2014 at 3:32 PM, Shivani Rao <raoshivani@gmail.com>wrote:
>>
>>> Hello folks,
>>>
>>> I was going to post this question to spark user group as well. If you
>>> have any leads on how to solve this issue please let me know:
>>>
>>> I am trying to build a basic spark project (spark depends on akka) and I
>>> am trying to create a fatjar using sbt assembly. The goal is to run the
>>> fatjar via commandline as follows:
>>>  java -cp "path to my spark fatjar" mainclassname
>>>
>>> I encountered deduplication errors in the following akka libraries
>>> during sbt assembly
>>> akka-remote_2.10-2.2.3.jar with
>>> akka-remote_2.10-2.2.3-shaded-protobuf.jar
>>>  akka-actor_2.10-2.2.3.jar with akka-actor_2.10-2.2.3-shaded-protobuf.jar
>>>
>>> I resolved them by using MergeStrategy.first and that helped with a
>>> successful compilation of the sbt assembly command. But for some or the
>>> other configuration parameter in the akka kept throwing up with the
>>> following message
>>>
>>> "Exception in thread "main" com.typesafe.config.ConfigException$Missing:
>>> No configuration setting found for key"
>>>
>>> I then used MergeStrategy.concat for "reference.conf" and I started
>>> getting this repeated error
>>>
>>> Exception in thread "main" com.typesafe.config.ConfigException$Missing:
>>> No configuration setting found for key 'akka.version'.
>>>
>>> I noticed that akka.version is only in the akka-actor jars and not in
>>> the akka-remote. The resulting reference.conf (in my final fat jar) does
>>> not contain akka.version either. So the strategy is not working.
>>>
>>> There are several things I could try
>>>
>>> a) Use the following dependency https://github.com/sbt/sbt-proguard
>>> b) Write a build.scala to handle merging of reference.conf
>>>
>>> https://spark-project.atlassian.net/browse/SPARK-395
>>>
>>> http://letitcrash.com/post/21025950392/howto-sbt-assembly-vs-reference-conf
>>>
>>> c) Create a reference.conf by merging all akka configurations and then
>>> passing it in my java -cp command as shown below
>>>
>>> java -cp <jar-name> -DConfig.file=<config>
>>>
>>> The main issue is that if I run the spark jar as "sbt run" there are no
>>> errors in accessing any of the akka configuration parameters. It is only
>>> when I run it via command line (java -cp <jar-name> classname) that I
>>> encounter the error.
>>>
>>> Which of these is a long term fix to akka issues? For now, I removed the
>>> akka dependencies and that solved the problem, but I know that is not a
>>> long term solution
>>>
>>> Regards,
>>> Shivani
>>>
>>> --
>>> Software Engineer
>>> Analytics Engineering Team@ Box
>>> Mountain View, CA
>>>
>>
>>
>
>
> --
> Software Engineer
> Analytics Engineering Team@ Box
> Mountain View, CA
>

Mime
View raw message