spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Schweigert (Jira)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-28921) Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10)
Date Fri, 30 Aug 2019 14:30:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-28921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16919596#comment-16919596
] 

Paul Schweigert commented on SPARK-28921:
-----------------------------------------

Longer-term solution will be to upgrade the version of the kubernetes-client : [https://github.com/fabric8io/kubernetes-client/pull/1669]

> Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10)
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-28921
>                 URL: https://issues.apache.org/jira/browse/SPARK-28921
>             Project: Spark
>          Issue Type: Bug
>          Components: Kubernetes
>    Affects Versions: 2.4.3
>            Reporter: Paul Schweigert
>            Priority: Minor
>
> Spark jobs are failing on latest versions of Kubernetes when jobs attempt to provision
executor pods (jobs like Spark-Pi that do not launch executors run without a problem):
>  
> Here's an example error message:
>  
> {code:java}
> 19/08/30 01:29:09 INFO ExecutorPodsAllocator: Going to request 2 executors from Kubernetes.
> 19/08/30 01:29:09 INFO ExecutorPodsAllocator: Going to request 2 executors from Kubernetes.19/08/30
01:29:09 WARN WatchConnectionManager: Exec Failure: HTTP 403, Status: 403 - 
> java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden' 
>     at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:216) 
>     at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:183) 
>     at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141) 
>     at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) 
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

>     at java.lang.Thread.run(Thread.java:748)
> {code}
>  
> Looks like the issue is caused by the internal master Kubernetes url not having the port
specified:
> [https://github.com/apache/spark/blob/master//resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala#L82:7]
>  
> Using the master with the port (443) seems to fix the problem.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message