spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <>
Subject Re: Accessing Kerberos Secured HDFS Resources from Spark on Mesos
Date Fri, 26 Jun 2015 22:51:39 GMT
On Fri, Jun 26, 2015 at 3:44 PM, Dave Ariens <> wrote:

>  Fair. I will look into an alternative with a generated delegation token.
>   However the same issue exists.   How can I have the executor run some
> arbitrary code when it gets a task assignment and before it proceeds to
> process it's resources?

Hmm, good question. If it doesn't already, Mesos could have its own
implementation of CoarseGrainedExecutorBackend that provides that
functionality. The only difference is that you'd run something before the
executor starts up, not before each task.

YARN actually doesn't do it that way; YARN provides the tokens to the
executor before the process starts, so that when you call
"UserGroupInformation.getCurrentUser()" the tokens are already there.

One way of doing that is by writing the tokens to a file and setting the
KRB5CCNAME env variable when starting the process. You can check the Hadoop
sources for details. Not sure if there's another way.

>     *From: *Marcelo Vanzin
> *Sent: *Friday, June 26, 2015 6:20 PM
> *To: *Dave Ariens
> *Cc: *Tim Chen; Olivier Girardot;
> *Subject: *Re: Accessing Kerberos Secured HDFS Resources from Spark on
> Mesos
>   On Fri, Jun 26, 2015 at 3:09 PM, Dave Ariens <>
> wrote:
>>  Would there be any way to have the task instances in the slaves call
>> the UGI login with a principal/keytab provided to the driver?
>  That would only work with a very small number of executors. If you have
> many login requests in a short period of time with the same principal, the
> KDC will start to deny logins. That's why delegation tokens are used
> instead of explicit logins.
>  --
> Marcelo


View raw message