hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11151) failed to create (put, copyFromLocal, cp, etc.) file in encryption zone after one day running
Date Thu, 02 Oct 2014 18:02:37 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156893#comment-14156893

Arun Suresh commented on HADOOP-11151:

[~andrew.wang], thanks for the followup review

So, about the volatile authToken, the KMSClientProvider also starts background threads to
fill the encryptedKeyCache when it goes below threshold, which is an HTTP call to the server
using the authToken, so if in case the authToken gets flushed, I wanted to avoid a situation
when the bg thread might use the older token

wrt the new if statement.. 
in kerberos mode, the first time the Client connects,  it initiates a SPNEGO handshake with
the server. This involves first sending an {{OPTIONS}} request, to which the Server will respond
with {{Set-Cookie:}} containing the correct {{hadoop.auth}} cookie. This cookie string is
then *set* in the authToken. All subsequent requests {{GET / POST / DELETE etc.}} from the
client would just send the {{hadoop.auth}} cookie. The issue is that the authToken can be
*set* only during initialization, i.e. the response to the OPTIONS request (you can blame
{{KerberosAuthenticator}} for that). Now, if the authToken is stale, the Server will send
a 401 to the client with {{WWW-Authenticate : Negotiate}} Header. This is intercepted by the
JDK SPNEGO implementing classes on the client side and it automatically responds (no control
is returned to the call method in KMSClientProvider) with an {{Authorize : Negotiate <kerb
token>}} header.  This will generally be accepted by the server and along with the 'success'
response, the KMS server also sends back a {{Set-Cookie}} with the correct value of {{hadoop.auth}}
cookie.. unfortunately, since this is not a response to a connection initialization call,
and since the authToken is already *set*, the new {{hadoop.auth}} cookie is simply ignored..
Which mean for every subsequent request after the authToken has gone stale, the client will
actually be sending 2 requests.. one with the old stale token which returns 401 and the second
which is successful. The only way to know in the {{KMSClientProvider}} that there was an authentication
failure and that the authToken needs to be flushed is to see if a SUCCESSFUL response has
a {{hadoop.auth}} Set-Cookie. Also since I cannot just 'set' the authToken with the cookie,
since the {{set()}} method is package protected.. only option I have is to create a new instance
of the authToken.. so that the next request from the client will initialize the SPNEGO handshake
and 'set' the authToken correctly.

Hope this made sense..

> failed to create (put, copyFromLocal, cp, etc.) file in encryption zone after one day
> ---------------------------------------------------------------------------------------------
>                 Key: HADOOP-11151
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11151
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 2.6.0
>            Reporter: zhubin
>            Assignee: Arun Suresh
>         Attachments: HADOOP-11151.1.patch, HADOOP-11151.2.patch, HADOOP-11151.3.patch
> Enable CFS and KMS service in the cluster, initially it worked to put/copy file into
encryption zone. But after a while (might be one day), it fails to put/copy file into the
encryption zone with the error
> java.util.concurrent.ExecutionException: java.io.IOException: HTTP status [403], message
> The kms.log shows below
> AbstractDelegationTokenSecretManager - Updating the current master key for generating
delegation tokens
> 2014-09-29 13:18:46,599 WARN  AuthenticationFilter - AuthenticationToken ignored: org.apache.hadoop.security.authentication.util.SignerException:
Invalid signature
> 2014-09-29 13:18:46,599 WARN  AuthenticationFilter - Authentication exception: Anonymous
requests are disallowed
> org.apache.hadoop.security.authentication.client.AuthenticationException: Anonymous requests
are disallowed
>         at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:184)
>         at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:331)
>         at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:507)
>         at org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:129)
>         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>         at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>         at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>         at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>         at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
>         at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>         at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
>         at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
>         at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>         at java.lang.Thread.run(Thread.java:745)

This message was sent by Atlassian JIRA

View raw message