spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sora" <>
Subject How to use spark-on-k8s pod template?
Date Tue, 05 Nov 2019 11:37:23 GMT
Hi all,
I am looking for the usage about the spark-on-k8s pod template.
I want to set some toleration rules for the driver and executor pod.
I tried to set --conf spark.kubernetes.driver.podTemplateFile=/spark-pod-template.yaml but
didn't work.
The driver pod started without the toleration rules and stay pending because of no available
node.Could anyone please show me any usage?

The template file is below.
apiVersion: extensions/v1beta1
kind: Pod
        - effect: NoSchedule
          key: project
          operator: Equal
          value: name

My full command is below.
/opt/spark/bin/spark-submit --master k8s://https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT_443_TCP_PORT
--conf spark.kubernetes.driver.podTemplateFile='/spark-pod-template.yaml' \
--conf spark.kubernetes.executor.podTemplateFile='/spark-pod-template.yaml' \
--conf spark.scheduler.mode=FAIR \
--conf spark.driver.memory=2g \
--conf spark.driver.cores=1 \
--conf spark.executor.cores=1 \
--conf spark.executor.memory=1g \
--conf spark.executor.instances=4 \
--conf spark.kubernetes.container.image=job-image \
--conf spark.kubernetes.namespace=nc \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=sa \
--conf \
--conf spark.kubernetes.submission.waitAppCompletion=false \
--deploy-mode cluster \
--name job-name \
--class job.class job.jar job-args

View raw message