spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yinan Li <liyinan...@gmail.com>
Subject Re: Kubernetes: why use init containers?
Date Wed, 10 Jan 2018 22:16:56 GMT
> 1500 less lines of code trump all of the arguments given so far for
> what the init container might be a good idea.

We can also reduce the #lines of code by simply refactoring the code in
such as way that a lot of code can be shared between configuration of the
main container and that of the ini-container. Actually we have been
discussing this as one of the things to do right after the 2.3 release and
we do have a Jira ticket to track it. It's probably true that none of the
arguments we made are convincing enough, but we can not rule out the
benefits init-containers bring either.

Again, I would suggest we look at this more thoroughly post 2.3.

On Wed, Jan 10, 2018 at 2:06 PM, Marcelo Vanzin <vanzin@cloudera.com> wrote:

> On Wed, Jan 10, 2018 at 2:00 PM, Yinan Li <liyinan926@gmail.com> wrote:
> > I want to re-iterate on one point, that the init-container achieves a
> clear
> > separation between preparing an application and actually running the
> > application. It's a guarantee provided by the K8s admission control and
> > scheduling components that if the init-container fails, the main
> container
> > won't be run. I think this is definitely positive to have. In the case
> of a
> > Spark application, the application code and driver/executor code won't
> even
> > be run if the init-container fails to localize any of the dependencies
>
> That is also the case with spark-submit... (can't download
> dependencies -> spark-submit fails before running user code).
>
> > Note that we are not blindly opposing getting rid of the init-container,
> > it's just that there's still valid reasons to keep it for now
>
> I'll flip that around: I'm not against having an init container if
> it's serving a needed purpose, it's just that nobody is able to tell
> me what that needed purpose is.
>
> 1500 less lines of code trump all of the arguments given so far for
> what the init container might be a good idea.
>
> --
> Marcelo
>

Mime
View raw message