spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <van...@cloudera.com>
Subject Re: Kubernetes: why use init containers?
Date Wed, 10 Jan 2018 22:35:52 GMT
On Wed, Jan 10, 2018 at 2:30 PM, Yinan Li <liyinan926@gmail.com> wrote:
> 1. Retries of init-containers are automatically supported by k8s through pod
> restart policies. For this point, sorry I'm not sure how spark-submit
> achieves this.

Great, add that feature to spark-submit, everybody benefits, not just k8s.

> 2. The ability to use credentials that are not shared with the main
> containers.

Not sure what that achieves.

> 3. Not only the user code, but Spark internal code like Executor won't be
> run if the init-container fails.

Not sure what that achieves. Executor will fail if dependency download
fails, Spark driver will recover (and start a new executor if needed).

> 4. Easier to build tooling around k8s events/status of the init-container in
> case of failures as it's doing exactly one thing: downloading dependencies.

Again, I don't see what is all this hoopla about fine grained control
of dependency downloads. Spark solved this years ago for Spark
applications. Don't reinvent the wheel.

-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message