spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <>
Subject Re: Kubernetes: why use init containers?
Date Fri, 12 Jan 2018 17:02:02 GMT
On Fri, Jan 12, 2018 at 4:13 AM, Eric Charles <> wrote:
>> Again, I don't see what is all this hoopla about fine grained control
>> of dependency downloads. Spark solved this years ago for Spark
>> applications. Don't reinvent the wheel.
> Init-containers are used today to download dependencies. I may be wrong and
> may open another facet of the discussion, but I see init container usage in
> a more generic way and not only restricted to dependencies download.

I'm not trying to discuss the general benefits of init containers as
it pertains to the kubernetes framework. I'm sure they added those for
a reason.

I'm trying to discuss them in the restricted scope of the spark-on-k8s
integration. And there, there is a single use for the single init
container that the Spark code itself injects into the pod: downloading
dependencies, which is something that spark-submit already does.

There's an option to override that one init container image with
another, where you can completely change its behavior. Given that
there is no contract currently that explains how these images should
behave, doing so is very, very risky and might break the application
completely (e.g. because dependencies are now not being downloaded, or
placed in the wrong location).

An you can do the exact same thing by overriding the main Spark image
itself. Just run the same code in your custom entry point before the
Spark-provided entry point runs. Same results and caveats as above

So again, the specific init container used by spark-on-k8s, as far as
I can see, seems to cause more problems than it solves.


To unsubscribe e-mail:

View raw message