spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4
Date Tue, 16 Oct 2018 19:48:16 GMT
Chiming in here. We are in the same boat as Bloomberg.

(But being a release manager often myself I understand the trade-off)

B.

Op di 16 okt. 2018 21:24 schreef Ilan Filonenko <if56@cornell.edu>:

> On Erik's note, would SPARK-23257 be included in, say, 2.4.1? When would
> the next RC be? I would like to propose the inclusion of the Kerberos
> feature sooner rather than later as it would increase Spark-on-K8S adoption
> in production workloads while bringing greater feature parity with Yarn and
> Mesos. I would like to note that the feature itself is isolated from Core
> and isolated via the step-based architecture of the Kubernetes
> Driver/Executor builders.
>
> Furthermore, Spark users traditionally use HDFS for storage and in
> production use-cases these HDFS clusters would be kerberized. At Bloomberg,
> for example, all of the HDFS clusters are kerberized and for this reason,
> the only thing stopping our internal Data Science Platform from adopting
> Spark-on-K8S is this feature.
>
> On Tue, Oct 16, 2018 at 10:21 AM Erik Erlandson <eerlands@redhat.com>
> wrote:
>
>>
>> SPARK-23257 merged more recently than I realized. If that isn't on
>> branch-2.4 then the first question is how soon on the release sequence that
>> can be adopted
>>
>> On Tue, Oct 16, 2018 at 9:33 AM Reynold Xin <rxin@databricks.com> wrote:
>>
>>> We shouldn’t merge new features into release branches anymore.
>>>
>>> On Tue, Oct 16, 2018 at 6:32 PM Rob Vesse <rvesse@dotnetrdf.org> wrote:
>>>
>>>> Right now the Kerberos support for Spark on K8S is only on master
>>>> AFAICT i.e. the feature is not present on branch-2.4
>>>>
>>>>
>>>>
>>>> Therefore I don’t see any point in adding the tests into branch-2.4
>>>> unless the plan is to also merge the Kerberos support to branch-2.4
>>>>
>>>>
>>>>
>>>> Rob
>>>>
>>>>
>>>>
>>>> *From: *Erik Erlandson <eerlands@redhat.com>
>>>> *Date: *Tuesday, 16 October 2018 at 16:47
>>>> *To: *dev <dev@spark.apache.org>
>>>> *Subject: *[DISCUSS][K8S][TESTS] Include Kerberos integration tests
>>>> for Spark 2.4
>>>>
>>>>
>>>>
>>>> I'd like to propose including integration testing for Kerberos on the
>>>> Spark 2.4 release:
>>>>
>>>> https://github.com/apache/spark/pull/22608
>>>>
>>>>
>>>>
>>>> Arguments in favor:
>>>>
>>>> 1) it improves testing coverage on a feature important for integrating
>>>> with HDFS deployments
>>>>
>>>> 2) its intersection with existing code is small - it consists primarily
>>>> of new testing code, with a bit of refactoring into 'main' and 'test'
>>>> sub-trees. These new tests appear stable.
>>>>
>>>> 3) Spark 2.4 is still in RC, with outstanding correctness issues.
>>>>
>>>>
>>>>
>>>> The argument 'against' that I'm aware of would be the relatively large
>>>> size of the PR. I believe this is considered above, but am soliciting
>>>> community feedback before committing.
>>>>
>>>> Cheers,
>>>>
>>>> Erik
>>>>
>>>>
>>>>
>>>

Mime
View raw message