spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saisai Shao <sai.sai.s...@gmail.com>
Subject Re: spark 2.0.0 - how to build an uber-jar?
Date Wed, 03 Aug 2016 08:54:29 GMT
I guess you're mentioning about spark assembly uber jar. In Spark 2.0,
there's no uber jar, instead there's a jars folder which contains all jars
required in the run-time. For the end user it is transparent, the way to
submit spark application is still the same.

On Wed, Aug 3, 2016 at 4:51 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com>
wrote:

> Just to clarify are you building Spark 2 from source downloaded?
>
> Or are you referring to building a uber jar file  using your code and mvn
> to submit with spark-submit etc?
>
> HTH
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 3 August 2016 at 09:43, lev <katzav@gmail.com> wrote:
>
>> hi,
>>
>> in spark 1.5, to build an uber-jar,
>> I would just compile the code with:
>> mvn ... package
>> and that will create one big jar with all the dependencies.
>>
>> when trying to do the same with spark 2.0, I'm getting a tar.gz file
>> instead.
>>
>> this is the full command I'm using:
>> mvn -Pyarn -Phive -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.4.7 -DskipTests
>> -Phive-thriftserver -Dscala-2.10 -Pbigtop-dist clean package
>>
>> how can I create the uber-jar?
>>
>> thanks.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-2-0-0-how-to-build-an-uber-jar-tp27463.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>>
>

Mime
View raw message