spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yinan Li <>
Subject Re: [Spark on Google Kubernetes Engine] Properties File Error
Date Mon, 30 Apr 2018 19:34:07 GMT
Also looks like you are mixing configuration properties from different
versions of Spark on Kubernetes.
"spark.kubernetes.{driver|executor}.docker.image" is only available in the
apache-spark-on-k8s fork, whereas "spark.kubernetes.container.image" is new
in Spark 2.3.0. Please make sure you use the same version of Spark to run
spark-submit and to build your image. You also need to make sure that you
use the right properties for the version of Spark you use. BTW: we suggest
people to use the official Spark 2.3.0 release instead of the fork. The
guide is at

On Mon, Apr 30, 2018 at 12:09 PM, Yinan Li <> wrote:

>  Which version of Spark are you using to run spark-submit, and which
> version of Spark your container image is based off? This looks to be caused
> my mismatched versions of Spark used for spark-submit and for the
> driver/executor at runtime.
> On Mon, Apr 30, 2018 at 12:00 PM, Holden Karau <>
> wrote:
>> So, while its not perfect, I have a guide focused on running custom Spark
>> on GKE
>> uture-apache-spark-releases-and-changes-on-google-kubernetes
>> -engine-and-cloud-dataproc and if you want to run pre-built Spark on GKE
>> there is a solutions article -
>> utions/spark-on-kubernetes-engine which could be relevant.
>> On Mon, Apr 30, 2018 at 7:51 PM, Eric Wang <>
>> wrote:
>>> Hello all,
>>> I've been trying to spark-submit a job to the Google Kubernetes Engine
>>> but I keep encountering a "Exception in thread "main"
>>> java.lang.IllegalArgumentException: Server properties file given at
>>> /opt/spark/work-dir/driver does not exist or is not a file."
>>> error. I'm unsure of how to even begin debugging this so any help would
>>> be greatly appreciated. I've attached the logs and the full spark-submit
>>> command I'm running here:
>>> erkkel/c04a0b5ca60ad755cf62e9ad18e5b7ed
>>> For reference, I've been following this guide: https://apache-spark-on
>>> Thanks,
>>> Eric
>> --
>> Twitter:

View raw message