spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Egor Pahomov <pahomov.e...@gmail.com>
Subject Re: Spark Assembly jar ?
Date Tue, 14 Jun 2016 22:21:20 GMT
It's strange for me, that having and support fat jar was never a important
thing. We have next scenario - we have big application, where spark is just
another library for data processing. So we can not create small jar and
feed it to spark scripts - we need to call spark from application. And
having fat jar as maven dependency is perfect. We have some spark installed
on cluster(whatever cloudera put there), but often we need to patch spark
for our needs, so we need to bring everything with us. Different
departments use different spark versions - so we can not share jars on
cluster easily. Yep, there are some disadvantages, but flexibility of
changing spark process and deploying overcome these disadvantages.

So we probably would patch pom's as usual to create fat jar.

2016-06-14 12:23 GMT-07:00 Reynold Xin <rxin@databricks.com>:

> You just need to run normal packaging and all the scripts are now setup to
> run without the assembly jars.
>
>
> On Tuesday, June 14, 2016, Franklyn D'souza <franklyn.dsouza@shopify.com>
> wrote:
>
>> Just wondering where the spark-assembly jar has gone in 2.0. i've been
>> reading that its been removed but i'm not sure what the new workflow is .
>>
>


-- 


*Sincerely yoursEgor Pakhomov*

Mime
View raw message