spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From akshay naidu <akshaynaid...@gmail.com>
Subject Re: Is Apache Spark-2.2.1 compatible with Hadoop-3.0.0
Date Mon, 08 Jan 2018 16:52:20 GMT
yes , spark download page does mention that 2.2.1 is for 'hadoop-2.7 and
later', but my confusion is because spark was released on 1st dec and
hadoop-3 stable version released on 13th Dec. And  to my similar question
on stackoverflow.com
<https://stackoverflow.com/questions/47920005/how-is-hadoop-3-0-0-s-compatibility-with-older-versions-of-hive-pig-sqoop-and>
, Mr. jacek-laskowski
<https://stackoverflow.com/users/1305344/jacek-laskowski> replied that
spark-2.2.1 doesn't support hadoop-3. so I am just looking for more clarity
on this doubt before moving on to upgrades.

Thanks all for help.
Akshay.

On Mon, Jan 8, 2018 at 8:47 AM, Saisai Shao <sai.sai.shao@gmail.com> wrote:

> AFAIK, there's no large scale test for Hadoop 3.0 in the community. So it
> is not clear whether it is supported or not (or has some issues). I think
> in the download page "Pre-Built for Apache Hadoop 2.7 and later" mostly
> means that it supports Hadoop 2.7+ (2.8...), but not 3.0 (IIUC).
>
> Thanks
> Jerry
>
> 2018-01-08 4:50 GMT+08:00 Raj Adyanthaya <rajady@gmail.com>:
>
>> Hi Akshay
>>
>> On the Spark Download page when you select Spark 2.2.1 it gives you an
>> option to select package type. In that, there is an option to select
>> "Pre-Built for Apache Hadoop 2.7 and later". I am assuming it means that it
>> does support Hadoop 3.0.
>>
>> http://spark.apache.org/downloads.html
>>
>> Thanks,
>> Raj A.
>>
>> On Sat, Jan 6, 2018 at 8:23 PM, akshay naidu <akshaynaidu.9@gmail.com>
>> wrote:
>>
>>> hello Users,
>>> I need to know whether we can run latest spark on  latest hadoop version
>>> i.e., spark-2.2.1 released on 1st dec and hadoop-3.0.0 released on 13th dec.
>>> thanks.
>>>
>>
>>
>

Mime
View raw message