spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dongjoon Hyun <>
Subject Use Hadoop-3.2 as a default Hadoop profile in 3.0.0?
Date Mon, 28 Oct 2019 19:33:29 GMT
Hi, All.

There was a discussion on publishing artifacts built with Hadoop 3 .
But, we are still publishing with Hadoop 2.7.3 and `3.0-preview` will be
the same because we didn't change anything yet.

Technically, we need to change two places for publishing.

1. Jenkins Snapshot Publishing

2. Release Snapshot/Release Publishing

To minimize the change, we need to switch our default Hadoop profile.

Currently, the default is `hadoop-2.7 (2.7.4)` profile and `hadoop-3.2
(3.2.0)` is optional.
We had better use `hadoop-3.2` profile by default and `hadoop-2.7`

Note that this means we use Hive 2.3.6 by default. Only `hadoop-2.7`
distribution will use `Hive 1.2.1` like Apache Spark 2.4.x.


View raw message