spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michel Sumbul <>
Subject Spark 3 pod template for the driver
Date Fri, 26 Jun 2020 12:53:00 GMT
Hi guys,
I try to use Spark 3 on top of Kubernetes and to specify a pod template for the driver.
Here is my pod manifest or the driver and when I do a spark-submit with the option:
--conf spark.kubernetes.driver.podTemplateFile=/data/k8s/podtemplate_driver3.yaml

I got the error message that I need to specify an image, but its the manifest.

Does my manifest file is wrong, how should it look like?

Thanks for your help,
--------The pod manifest:
apiVersion: v1kind: Podmetadata:  name: mySpark3App  labels:    app: mySpark3App   
customlabel/app-id: "1"spec:  securityContext:    runAsUser: 1000  volumes:    - name:
"test-volume"      emptyDir: {}  containers:    - name: spark3driver      image:      instances: 1      resources: 
      requests:          cpu: "1000m"          memory: "512Mi"        limits: 
        cpu: "1000m"          memory: "512Mi"      volumeMounts:       - name:
"test-volume"         mountPath: "/tmp"

View raw message