spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <>
Subject Re: Spark standalone mode and kerberized cluster
Date Tue, 16 Jun 2015 10:10:40 GMT

On 15 Jun 2015, at 15:43, Borja Garrido Bear <<>>

I tried running the job in a standalone cluster and I'm getting this: Failed on local exception:
Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "worker-node/<>";
destination host is: "hdfs":9000;

Both nodes can access the HDFS running spark locally, and have valid kerberos credentials,
I know for the moment keytab is not supported for standalone mode, but as long as the tokens
I had when initiating the workers and masters are valid this should work, shouldn't it?

I don't know anything about tokens on standalone. In YARN what we have to do is something
called "delegation tokens", the client asks (something) for tokens granting access to HDFS,
and attaches that to the YARN container creation request, which is then handed off to the
app master, which then gets to deal with (a) passing them down to launched workers and (b)
dealing with token refresh (which is where keytabs come in to play)

Why not try sshing in to the worker-node as the spark user and run kinit there to see if the
problem goes away once you've logged in with Kerberos. If that works, you're going to have
to automate that process across the cluster

View raw message