spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saisai Shao <sai.sai.s...@gmail.com>
Subject Re: Hadoop 3 support
Date Tue, 03 Apr 2018 00:30:53 GMT
Yes, the main blocking issue is the hive version used in Spark
(1.2.1.spark) doesn't support run on Hadoop 3. Hive will check the Hadoop
version in the runtime [1]. Besides this I think some pom changes should be
enough to support Hadoop 3.

If we want to use Hadoop 3 shaded client jar, then the pom requires lots of
changes, but this is not necessary.


[1]
https://github.com/apache/hive/blob/6751225a5cde4c40839df8b46e8d241fdda5cd34/shims/common/src/main/java/org/apache/hadoop/hive/shims/ShimLoader.java#L144

2018-04-03 4:57 GMT+08:00 Marcelo Vanzin <vanzin@cloudera.com>:

> Saisai filed SPARK-23534, but the main blocking issue is really
> SPARK-18673.
>
>
> On Mon, Apr 2, 2018 at 1:00 PM, Reynold Xin <rxin@databricks.com> wrote:
> > Does anybody know what needs to be done in order for Spark to support
> Hadoop
> > 3?
> >
>
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Mime
View raw message