spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Resolved] (SPARK-4491) Using sbt assembly with spark as dep requires Phd in sbt
Date Sat, 24 Jan 2015 18:42:34 GMT


Sean Owen resolved SPARK-4491.
    Resolution: Won't Fix

I don't see a reason here that indicates the Spark build should change its behavior. I do
agree that this build and its deps are hairy, and harmonizing dependencies within Spark and
with an app is often a deep rabbit hole.

> Using sbt assembly with spark as dep requires Phd in sbt
> --------------------------------------------------------
>                 Key: SPARK-4491
>                 URL:
>             Project: Spark
>          Issue Type: Question
>            Reporter: sam
> I get the dreaded deduplicate error from sbt.  I resolved the issue (I think, I managed
to run the SimpleApp example) here
> My question is, is this wise? What is wrong with changing the `deduplicate` bit to `first`.
 Why isn't it this by default?
> If this isn't the way to make it work, please could someone provide an explanation of
the correct way with .sbt examples. Having googled, every example I see is different because
it changes depending on what deps the person has ... surely there has to be an automagic way
of doing it (if my way isn't it)?
> One final point, SBT seems to be blaming Spark for causing the problem in their documentation: is this fair? Is Spark doing something wrong in the way
they build their jars? Or should SBT be renamed to CBT (Complicated Build Tool that will make
you need Cognitive Behavioural Therapy after use).
> NOTE: Satire JFF, really I love both SBT & Spark :)

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message