spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Bender <jonathan.ben...@gmail.com>
Subject Re: [VOTE] Release Apache Spark 1.5.0 (RC2)
Date Fri, 28 Aug 2015 22:18:10 GMT
Marcelo,

Thanks for replying -- after looking at my test again, I misinterpreted
another issue I'm seeing which is unrelated (note I'm not using a pre-built
binary, rather had to build my own with Yarn/Hive support, as I want to use
it on an older cluster (CDH5.1.0)).

I can start up a pyspark app on YARN, so I don't want to block this.  +1

Best,
Jonathan

On Fri, Aug 28, 2015 at 2:34 PM, Marcelo Vanzin <vanzin@cloudera.com> wrote:

> Hi Jonathan,
>
> Can you be more specific about what problem you're running into?
>
> SPARK-6869 fixed the issue of pyspark vs. assembly jar by shipping the
> pyspark archives separately to YARN. With that fix in place, pyspark
> doesn't need to get anything from the Spark assembly, so it has no
> problems running on YARN. I just downloaded
> spark-1.5.0-bin-hadoop2.6.tgz and tried that out, and pyspark works
> fine on YARN for me.
>
>
> On Fri, Aug 28, 2015 at 2:22 PM, Jonathan Bender
> <jonathan.bender@gmail.com> wrote:
> > -1 for regression on PySpark + YARN support
> >
> > It seems like this JIRA https://issues.apache.org/jira/browse/SPARK-7733
> > added a requirement for Java 7 in the build process.  Due to some quirks
> > with the Java archive format changes between Java 6 and 7, using PySpark
> > with a YARN uberjar seems to break when compiled with anything after
> Java 6
> > (see https://issues.apache.org/jira/browse/SPARK-1920 for reference).
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-5-0-RC2-tp13826p13890.html
> > Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > For additional commands, e-mail: dev-help@spark.apache.org
> >
>
>
>
> --
> Marcelo
>

Mime
View raw message