spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yinan Li <liyinan...@gmail.com>
Subject Re: Kubernetes backend and docker images
Date Sat, 06 Jan 2018 03:26:22 GMT
This is neat. With some code cleanup and as long as users can still use
custom driver/executor/init-container images if they want to, I think this
is great to have. I don't think there's a particular reason why having a
single image wouldn't work. Thanks for doing this!

On Fri, Jan 5, 2018 at 5:06 PM, Marcelo Vanzin <vanzin@cloudera.com> wrote:

> Hey all, especially those working on the k8s stuff.
>
> Currently we have 3 docker images that need to be built and provided
> by the user when starting a Spark app: driver, executor, and init
> container.
>
> When the initial review went by, I asked why do we need 3, and I was
> told that's because they have different entry points. That never
> really convinced me, but well, everybody wanted to get things in to
> get the ball rolling.
>
> But I still think that's not the best way to go. I did some pretty
> simple hacking and got things to work with a single image:
>
> https://github.com/vanzin/spark/commit/k8s-img
>
> Is there a reason why that approach would not work? You could still
> create separate images for driver and executor if wanted, but there's
> no reason I can see why we should need 3 images for the simple case.
>
> Note that the code there can be cleaned up still, and I don't love the
> idea of using env variables to propagate arguments to the container,
> but that works for now.
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Mime
View raw message