spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prashant Sharma <scrapco...@gmail.com>
Subject Re: Error while running Spark on K8s
Date Mon, 04 Jan 2021 13:14:40 GMT
Can you please paste the full exception trace, and mention spark and k8s
version?

On Mon, Jan 4, 2021 at 6:19 PM Sachit Murarka <connectsachit@gmail.com>
wrote:

> Hi Prashant
>
> Thanks for the response!
>
> I created the service account with the permissions  and following is the
> command:
>
> spark-submit --deploy-mode cluster --master k8s://http://ip:port --name
> "sachit"   --conf spark.kubernetes.pyspark.pythonVersion=3
> --conf spark.kubernetes.namespace=spark-test --conf
> spark.executor.instances=5 --conf
> spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa  --conf
> spark.kubernetes.container.image=sparkpy local:///opt/spark/da/main.py
>
> Kind Regards,
> Sachit Murarka
>
>
> On Mon, Jan 4, 2021 at 5:46 PM Prashant Sharma <scrapcodes@gmail.com>
> wrote:
>
>> Hi Sachit,
>>
>> Can you give more details on how did you run? i.e. spark submit command.
>> My guess is, a service account with sufficient privilege is not provided.
>> Please see:
>> http://spark.apache.org/docs/latest/running-on-kubernetes.html#rbac
>>
>> Thanks,
>>
>> On Mon, Jan 4, 2021 at 5:27 PM Sachit Murarka <connectsachit@gmail.com>
>> wrote:
>>
>>> Hi All,
>>> I am getting the below error when I am trying to run the spark job on
>>> Kubernetes, I am running it in cluster mode.
>>>
>>> Exception in thread "main"
>>> io.fabric8.kubernetes.client.KubernetesClientException: Operation:
>>> [create] for kind: [Pod] with name: [null] in namespace:
>>> [spark-test] failed.
>>> at
>>> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
>>> at
>>> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72)
>>>
>>>
>>> I saw there was an JIRA opened already. SPARK-31786
>>> I tried the parameter mentioned in JIRA
>>> too(spark.kubernetes.driverEnv.HTTP2_DISABLE=true), that also did not work.
>>> Can anyone suggest what can be done?
>>>
>>> Kind Regards,
>>> Sachit Murarka
>>>
>>

Mime
View raw message