spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: HDP 2.3 support for Spark 1.5.x
Date Mon, 28 Sep 2015 16:47:00 GMT
Krishna:
If you want to query ORC files, see the following JIRA:

[SPARK-10623] [SQL] Fixes ORC predicate push-down

which is in the 1.5.1. release.

FYI

On Mon, Sep 28, 2015 at 9:42 AM, Fabien Martin <fabien.martin78@gmail.com>
wrote:

> Hi Krishna,
>
>    - Take a lokk at *http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/
>    <http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/>*
>    - Or you can specify your 1.5.x jar as the Spark one using something
>    like :
>
> --conf
> "spark.yarn.jar=hdfs://master:8020/spark-assembly-1.5.0-hadoop2.6.0.jar"
>
> The main drawback is :
>
> *Known Issues*
>
> *Spark YARN ATS integration does not work in this tech preview. You will
> not see the history of Spark jobs in the Jobs server after a job is
> finished.*
>
> 2015-09-23 1:31 GMT+02:00 Zhan Zhang <zzhang@hortonworks.com>:
>
>> Hi Krishna,
>>
>> For the time being, you can download from upstream, and it should be
>> running OK for HDP2.3.  For hdp specific problem, you can ask in
>> Hortonworks forum.
>>
>> Thanks.
>>
>> Zhan Zhang
>>
>> On Sep 22, 2015, at 3:42 PM, Krishna Sankar <ksankar42@gmail.com> wrote:
>>
>> Guys,
>>
>>    - We have HDP 2.3 installed just now. It comes with Spark 1.3.x. The
>>    current wisdom is that it will support the 1.4.x train (which is good, need
>>    DataFrame et al).
>>    - What is the plan to support Spark 1.5.x ? Can we install 1.5.0 on
>>    HDP 2.3 ? Or will Spark 1.5.x support be in HDP 2.3.x and if so ~when ?
>>
>> Cheers & Thanks
>> <k/>
>>
>>
>>
>

Mime
View raw message