spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Efe Selcuk <efema...@gmail.com>
Subject Re: Spark2 SBT Assembly
Date Thu, 11 Aug 2016 18:29:06 GMT
Bump!

On Wed, Aug 10, 2016 at 2:59 PM, Efe Selcuk <efeman92@gmail.com> wrote:

> Thanks for the replies, folks.
>
> My specific use case is maybe unusual. I'm working in the context of the
> build environment in my company. Spark was being used in such a way that
> the fat assembly jar that the old 'sbt assembly' command outputs was used
> when building a spark applicaiton. I'm trying to figure out if I can just
> use the many library jars instead, but in the meantime I'm hoping to get a
> fat assembly in the old way to get us unblocked in updating our application
> to use 2.0. It's a proprietary build system, not maven or sbt, so it's not
> straightforward and the dependencies are modeled differently.
>
> To be a bit more clear: the fat assembly was not used for any reason other
> than to get the spark application to build. This is in the context of
> running in Amazon EMR, so we don't send that spark assembly over for runs.
>
> Efe
>
> On Wed, Aug 10, 2016 at 2:15 PM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>> Hi Efe,
>>
>> Are you talking about creating an uber/fat jar file for your specific
>> application? Then you can distribute it to another node just to use the jar
>> file without assembling it.
>>
>> I can still do it in Spark 2 as before if I understand your special use
>> case.
>>
>> [warn] Strategy 'discard' was applied to 349 files
>> [warn] Strategy 'first' was applied to 450 files
>>
>> *[info] Assembly up to date:
>> /data6/hduser/scala/CEP_streaming/target/scala-2.10/scala-assembly-1.0.jar*[success]
>> Total time: 117 s, completed Aug 10, 2016 9:31:24 PM
>> Submiting the job
>> Ivy Default Cache set to: /home/hduser/.ivy2/cache
>> The jars for the packages stored in: /home/hduser/.ivy2/jars
>>
>>
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 10 August 2016 at 20:35, Holden Karau <holden@pigscanfly.ca> wrote:
>>
>>> What are you looking to use the assembly jar for - maybe we can think of
>>> a workaround :)
>>>
>>>
>>> On Wednesday, August 10, 2016, Efe Selcuk <efeman92@gmail.com> wrote:
>>>
>>>> Sorry, I should have specified that I'm specifically looking for that
>>>> fat assembly behavior. Is it no longer possible?
>>>>
>>>> On Wed, Aug 10, 2016 at 10:46 AM, Nick Pentreath <
>>>> nick.pentreath@gmail.com> wrote:
>>>>
>>>>> You're correct - Spark packaging has been shifted to not use the
>>>>> assembly jar.
>>>>>
>>>>> To build now use "build/sbt package"
>>>>>
>>>>>
>>>>>
>>>>> On Wed, 10 Aug 2016 at 19:40, Efe Selcuk <efeman92@gmail.com> wrote:
>>>>>
>>>>>> Hi Spark folks,
>>>>>>
>>>>>> With Spark 1.6 the 'assembly' target for sbt would build a fat jar
>>>>>> with all of the main Spark dependencies for building an application.
>>>>>> Against Spark 2, that target is no longer building a spark assembly,
just
>>>>>> ones for e.g. Flume and Kafka.
>>>>>>
>>>>>> I'm not well versed with maven and sbt, so I don't know how to go
>>>>>> about figuring this out.
>>>>>>
>>>>>> Is this intended? Or am I missing something?
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>
>>>>
>>>
>>> --
>>> Cell : 425-233-8271
>>> Twitter: https://twitter.com/holdenkarau
>>>
>>>
>>
>

Mime
View raw message