spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erik Erlandson (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
Date Fri, 01 Jun 2018 17:54:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16498337#comment-16498337
] 

Erik Erlandson edited comment on SPARK-24434 at 6/1/18 5:53 PM:
----------------------------------------------------------------

My current take on UX around this feature is that there's not much precedent from the Spark
world. Assuming I'm right about that it's more likely to be driven by what expectations Kubernetes
users have. In my experience that is along the lines of "pointing at a yaml file," but maybe
there's more variety of user workflows than I think.

JSON definitely seems more amenable for including with command-line arguments. I have been
assuming that if users were specifying pod configurations that they'd be somewhat larger pod
sub-structures and not easy to supply embedded on a command line. Are "small" pod modifications
also likely?


was (Author: eje):
My current take on UX around this feature is that there's not much precedent from the Spark
world. Assuming I'm right about that it's more likely to be driven by what expectations Kubernetes
users have. In my experience that is along the lines of "pointing at a yaml file," but maybe
there's more variety of user workflows than I think.

> Support user-specified driver and executor pod templates
> --------------------------------------------------------
>
>                 Key: SPARK-24434
>                 URL: https://issues.apache.org/jira/browse/SPARK-24434
>             Project: Spark
>          Issue Type: New Feature
>          Components: Kubernetes
>    Affects Versions: 2.4.0
>            Reporter: Yinan Li
>            Priority: Major
>
> With more requests for customizing the driver and executor pods coming, the current approach
of adding new Spark configuration options has some serious drawbacks: 1) it means more Kubernetes
specific configuration options to maintain, and 2) it widens the gap between the declarative
model used by Kubernetes and the configuration model used by Spark. We should start designing
a solution that allows users to specify pod templates as central places for all customization
needs for the driver and executor pods. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message