spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rob Vesse (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-26833) Kubernetes RBAC documentation is unclear on exact RBAC requirements
Date Wed, 06 Feb 2019 10:44:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-26833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rob Vesse updated SPARK-26833:
------------------------------
    Description: 
I've seen a couple of users get bitten by this in informal discussions on GitHub and Slack.
 Basically the user sets up the service account and configures Spark to use it as described
in the documentation but then when they try and run a job they encounter an error like the
following:

{noformat}
019-02-05 20:29:02 WARN  WatchConnectionManager:185 - Exec Failure: HTTP 403, Status: 403
- pods "spark-pi-1549416541302-driver" is forbidden: User "system:anonymous" cannot watch
pods in the namespace "default"
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
...
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: pods "spark-pi-1549416541302-driver"
is forbidden: User "system:anonymous" cannot watch pods in the namespace "default"
{noformat}

This error stems from the fact that the configured service account is only used by the driver
pod and not by the submission client.  The submission client wants to do driver pod monitoring
which it does with the users submission credentials *NOT* the service account as the user
might expect.

It seems like there are two ways to resolve this issue:

* Improve the documentation to clarify the current situation
* Ensure that if a service account is configured we always use it even on the submission client

The former is the easy fix, the latter is more invasive and may have other knock on effects
so we should start with the former and discuss the feasibility of the latter.

  was:
I've seen a couple of users get bitten by this in informal discussions on GitHub and Slack.
 Basically the user sets up the service account and configures Spark to use it as described
in the documentation but then when they try and run a job they encounter an error like the
following:

{noformat}
019-02-05 20:29:02 WARN  WatchConnectionManager:185 - Exec Failure: HTTP 403, Status: 403
- pods "spark-pi-1549416541302-driver" is forbidden: User "system:anonymous" cannot watch
pods in the namespace "default"
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
...
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: pods "spark-pi-1549416541302-driver"
is forbidden: User "system:anonymous" cannot watch pods in the namespace "default"
{noformat}

This error stems from the fact that the configured service account is only used by the driver
pod and not by the submission client.  The submission client wants to do driver pod monitoring
which it does with the users submission credentials **NOT** the service account as the user
might expect.

It seems like there are two ways to resolve this issue:

* Improve the documentation to clarify the current situation
* Ensure that if a service account is configured we always use it even on the submission client

The former is the easy fix, the latter is more invasive and may have other knock on effects
so we should start with the former and discuss the feasibility of the latter.


> Kubernetes RBAC documentation is unclear on exact RBAC requirements
> -------------------------------------------------------------------
>
>                 Key: SPARK-26833
>                 URL: https://issues.apache.org/jira/browse/SPARK-26833
>             Project: Spark
>          Issue Type: Improvement
>          Components: Kubernetes
>    Affects Versions: 2.3.0, 2.3.1, 2.3.2, 2.4.0
>            Reporter: Rob Vesse
>            Priority: Major
>
> I've seen a couple of users get bitten by this in informal discussions on GitHub and
Slack.  Basically the user sets up the service account and configures Spark to use it as described
in the documentation but then when they try and run a job they encounter an error like the
following:
> {noformat}
> 019-02-05 20:29:02 WARN  WatchConnectionManager:185 - Exec Failure: HTTP 403, Status:
403 - pods "spark-pi-1549416541302-driver" is forbidden: User "system:anonymous" cannot watch
pods in the namespace "default"
> java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
> ...
> Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: pods
"spark-pi-1549416541302-driver" is forbidden: User "system:anonymous" cannot watch pods in
the namespace "default"
> {noformat}
> This error stems from the fact that the configured service account is only used by the
driver pod and not by the submission client.  The submission client wants to do driver pod
monitoring which it does with the users submission credentials *NOT* the service account as
the user might expect.
> It seems like there are two ways to resolve this issue:
> * Improve the documentation to clarify the current situation
> * Ensure that if a service account is configured we always use it even on the submission
client
> The former is the easy fix, the latter is more invasive and may have other knock on effects
so we should start with the former and discuss the feasibility of the latter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message