spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamas Sandor <tsan...@gmail.com>
Subject Re: submitted uber-jar not seeing spark-assembly.jar at worker
Date Thu, 16 Oct 2014 11:32:12 GMT
Hello Owen,

I used maven build to make use of the guava collections package renaming,
sbt keeps the old Guava package names intact...
Finally it turned out that I have just upgraded to the latest version of
spark-cassandra-connector: 1.1.0-alpha3 and when I step back to
1.1.0-alpha2 everything started to work again... though I needed the manual
build with the Guava-conflict solved.

Thanks for you help.


On Wed, Oct 15, 2014 at 9:01 AM, Sean Owen <sowen@cloudera.com> wrote:

> How did you recompile and deploy Spark to your cluster? it sounds like
> a problem with not getting the assembly deployed correctly, rather
> than your app.
>
> On Tue, Oct 14, 2014 at 10:35 PM, Tamas Sandor <tsandor@gmail.com> wrote:
> > Hi,
> >
> > I'm rookie in spark, but hope someone can help me out. I'm writing an app
> > that I'm submitting to my spark-master that has a worker on a separate
> node.
> > It uses spark-cassandra-connector, and since it depends on guava-v16 and
> it
> > conflicts with the default spark-1.1.0-assembly's guava-v14.1 I built the
> > latest from spark git master (it was fixed in late Sept), so now I have a
> > working  spark-assembly-1.2.0-SNAPSHOT-hadoop2.4.0 running.
> >
> > I have my uber-jar that has hadoop-client and spark-assembly as
> > scope:provided, excluded from the deployed jar and than it gets
> submitted to
> > a spark-master from the node. From the logs I see  taskSetManager throws
> me
> > an error coming from my worker node saying
> > "java.lang.NoClassDefFoundError:org/apache/spark/Partition" - I guess
> valid
> > since my jar has no spark deps inline (uber) but why it cannot see the
> > workers classpath - this what a "provided" scope would mean here?
> >
> > How can I fix that? Am I missing something obvious?
> > Thank you for your help.
> >
> > Regards,
> > Tamas
>

Mime
View raw message