spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabien Martin <fabien.marti...@gmail.com>
Subject Re: HDP 2.3 support for Spark 1.5.x
Date Mon, 28 Sep 2015 16:42:49 GMT
Hi Krishna,

   - Take a lokk at
*http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/
   <http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/>*
   - Or you can specify your 1.5.x jar as the Spark one using something
   like :

--conf
"spark.yarn.jar=hdfs://master:8020/spark-assembly-1.5.0-hadoop2.6.0.jar"

The main drawback is :

*Known Issues*

*Spark YARN ATS integration does not work in this tech preview. You will
not see the history of Spark jobs in the Jobs server after a job is
finished.*

2015-09-23 1:31 GMT+02:00 Zhan Zhang <zzhang@hortonworks.com>:

> Hi Krishna,
>
> For the time being, you can download from upstream, and it should be
> running OK for HDP2.3.  For hdp specific problem, you can ask in
> Hortonworks forum.
>
> Thanks.
>
> Zhan Zhang
>
> On Sep 22, 2015, at 3:42 PM, Krishna Sankar <ksankar42@gmail.com> wrote:
>
> Guys,
>
>    - We have HDP 2.3 installed just now. It comes with Spark 1.3.x. The
>    current wisdom is that it will support the 1.4.x train (which is good, need
>    DataFrame et al).
>    - What is the plan to support Spark 1.5.x ? Can we install 1.5.0 on
>    HDP 2.3 ? Or will Spark 1.5.x support be in HDP 2.3.x and if so ~when ?
>
> Cheers & Thanks
> <k/>
>
>
>

Mime
View raw message