spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Erlandson <>
Subject Re: [k8s] Spark operator (the Java one)
Date Sat, 19 Oct 2019 17:31:07 GMT
> It's applicable regardless of if the operators are maintained as part of
> Spark core or not, with the maturity of Kubernetes features around CRD
> support and webhooks. The GCP Spark operator supports a lot of additional
> pod/container configs using a webhook, and this approach seems pretty
> successful so far.

Agreed (in fact the existence of >= two independent operator projects
testifies to this). I do believe this has implications for how feature
requests for spark-on-k8s get fielded here upstream. There's a non-zero
amount of cognitive load involved with recommending that a feature request
be deferred to some independent operator project. Going forward, will that
complicate the upstream story for spark-submit, history server, and shuffle
service on a kube backend?

View raw message