spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sachit Murarka <connectsac...@gmail.com>
Subject Re: Error while running Spark on K8s
Date Mon, 04 Jan 2021 12:48:56 GMT
Hi Prashant

Thanks for the response!

I created the service account with the permissions  and following is the
command:

spark-submit --deploy-mode cluster --master k8s://http://ip:port --name
"sachit"   --conf spark.kubernetes.pyspark.pythonVersion=3
--conf spark.kubernetes.namespace=spark-test --conf
spark.executor.instances=5 --conf
spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa  --conf
spark.kubernetes.container.image=sparkpy local:///opt/spark/da/main.py

Kind Regards,
Sachit Murarka


On Mon, Jan 4, 2021 at 5:46 PM Prashant Sharma <scrapcodes@gmail.com> wrote:

> Hi Sachit,
>
> Can you give more details on how did you run? i.e. spark submit command.
> My guess is, a service account with sufficient privilege is not provided.
> Please see:
> http://spark.apache.org/docs/latest/running-on-kubernetes.html#rbac
>
> Thanks,
>
> On Mon, Jan 4, 2021 at 5:27 PM Sachit Murarka <connectsachit@gmail.com>
> wrote:
>
>> Hi All,
>> I am getting the below error when I am trying to run the spark job on
>> Kubernetes, I am running it in cluster mode.
>>
>> Exception in thread "main"
>> io.fabric8.kubernetes.client.KubernetesClientException: Operation:
>> [create] for kind: [Pod] with name: [null] in namespace:
>> [spark-test] failed.
>> at
>> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
>> at
>> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72)
>>
>>
>> I saw there was an JIRA opened already. SPARK-31786
>> I tried the parameter mentioned in JIRA
>> too(spark.kubernetes.driverEnv.HTTP2_DISABLE=true), that also did not work.
>> Can anyone suggest what can be done?
>>
>> Kind Regards,
>> Sachit Murarka
>>
>

Mime
View raw message