spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <sro...@gmail.com>
Subject Re: Use Hadoop-3.2 as a default Hadoop profile in 3.0.0?
Date Mon, 28 Oct 2019 19:43:56 GMT
I'm OK with that, but don't have a strong opinion nor info about the
implications.
That said my guess is we're close to the point where we don't need to
support Hadoop 2.x anyway, so, yeah.

On Mon, Oct 28, 2019 at 2:33 PM Dongjoon Hyun <dongjoon.hyun@gmail.com> wrote:
>
> Hi, All.
>
> There was a discussion on publishing artifacts built with Hadoop 3 .
> But, we are still publishing with Hadoop 2.7.3 and `3.0-preview` will be the same because
we didn't change anything yet.
>
> Technically, we need to change two places for publishing.
>
> 1. Jenkins Snapshot Publishing
>     https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-maven-snapshots/
>
> 2. Release Snapshot/Release Publishing
>     https://github.com/apache/spark/blob/master/dev/create-release/release-build.sh
>
> To minimize the change, we need to switch our default Hadoop profile.
>
> Currently, the default is `hadoop-2.7 (2.7.4)` profile and `hadoop-3.2 (3.2.0)` is optional.
> We had better use `hadoop-3.2` profile by default and `hadoop-2.7` optionally.
>
> Note that this means we use Hive 2.3.6 by default. Only `hadoop-2.7` distribution will
use `Hive 1.2.1` like Apache Spark 2.4.x.
>
> Bests,
> Dongjoon.

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message