spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anirudh Ramanathan <ramanath...@google.com.INVALID>
Subject Re: handling Remote dependencies for spark-submit in spark 2.3 with kubernetes
Date Thu, 08 Mar 2018 20:05:47 GMT
You don't need to create the init-container. It's an implementation detail.
If you provide a remote uri, and
specify spark.kubernetes.container.image=<spark-image>, Spark *internally*
will add the init container to the pod spec for you.
*If *for some reason, you want to customize the init container image, you
can choose to do that using the specific options, but I don't think this is
necessary in most scenarios. The init container image, driver and executor
images can be identical by default.


On Thu, Mar 8, 2018 at 6:52 AM purna pradeep <purna2pradeep@gmail.com>
wrote:

> Im trying to run spark-submit to kubernetes cluster with spark 2.3 docker
> container image
>
> The challenge im facing is application have a mainapplication.jar and
> other dependency files & jars which are located in Remote location like AWS
> s3 ,but as per spark 2.3 documentation there is something called kubernetes
> init-container to download remote dependencies but in this case im not
> creating any Podspec to include init-containers in kubernetes, as per
> documentation Spark 2.3 spark/kubernetes internally creates Pods
> (driver,executor) So not sure how can i use init-container for spark-submit
> when there are remote dependencies.
>
>
> https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-remote-dependencies
>
> Please suggest
>


-- 
Anirudh Ramanathan

Mime
View raw message