spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yinan Li (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
Date Fri, 01 Jun 2018 17:40:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16498307#comment-16498307
] 

Yinan Li edited comment on SPARK-24434 at 6/1/18 5:39 PM:
----------------------------------------------------------

The pod template is basically a pod specification and can contain every possible pieces of
information about a pod. It should look similar to what the core workload types (deployments
and statefulsets for example) use, which contains a {{[PodSpec|https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/core/v1/types.go#L2636]}}.  

The problem is unique for the Kubernetes mode as there are many things to customize for a
pod. Currently we basically just introduce a new Spark config property for each new customization
aspect of a pod. Given the number of things to customize, this will soon become hard to maintain
if we keep introducing new config properties. 


was (Author: liyinan926):
The pod template is basically a pod specification and can contain every possible pieces of
information about a pod. It should look similar to what the core workload types (deployments
and statefulsets for example) use, which contains a {{[PodSpec|https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/core/v1/types.go#L2636]}}. 

 

The problem is unique for the Kubernetes mode as there are many things to customize for a
pod. Currently we basically just introduce a new Spark config property for each new customization
aspect of a pod. Given the number of things to customize, this will soon become hard to maintain
if we keep introducing new config properties. 

> Support user-specified driver and executor pod templates
> --------------------------------------------------------
>
>                 Key: SPARK-24434
>                 URL: https://issues.apache.org/jira/browse/SPARK-24434
>             Project: Spark
>          Issue Type: New Feature
>          Components: Kubernetes
>    Affects Versions: 2.4.0
>            Reporter: Yinan Li
>            Priority: Major
>
> With more requests for customizing the driver and executor pods coming, the current approach
of adding new Spark configuration options has some serious drawbacks: 1) it means more Kubernetes
specific configuration options to maintain, and 2) it widens the gap between the declarative
model used by Kubernetes and the configuration model used by Spark. We should start designing
a solution that allows users to specify pod templates as central places for all customization
needs for the driver and executor pods. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message