spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Chen <...@mesosphere.io>
Subject Re: Accessing Kerberos Secured HDFS Resources from Spark on Mesos
Date Sat, 27 Jun 2015 06:56:37 GMT
Does YARN provide the token through that env variable you mentioned? Or how
does YARN do this?

Tim

On Fri, Jun 26, 2015 at 3:51 PM, Marcelo Vanzin <vanzin@cloudera.com> wrote:

> On Fri, Jun 26, 2015 at 3:44 PM, Dave Ariens <dariens@blackberry.com>
> wrote:
>
>>  Fair. I will look into an alternative with a generated delegation
>> token.   However the same issue exists.   How can I have the executor run
>> some arbitrary code when it gets a task assignment and before it proceeds
>> to process it's resources?
>>
>
> Hmm, good question. If it doesn't already, Mesos could have its own
> implementation of CoarseGrainedExecutorBackend that provides that
> functionality. The only difference is that you'd run something before the
> executor starts up, not before each task.
>
> YARN actually doesn't do it that way; YARN provides the tokens to the
> executor before the process starts, so that when you call
> "UserGroupInformation.getCurrentUser()" the tokens are already there.
>
> One way of doing that is by writing the tokens to a file and setting the
> KRB5CCNAME env variable when starting the process. You can check the Hadoop
> sources for details. Not sure if there's another way.
>
>
>
>>
>>     *From: *Marcelo Vanzin
>> *Sent: *Friday, June 26, 2015 6:20 PM
>> *To: *Dave Ariens
>> *Cc: *Tim Chen; Olivier Girardot; user@spark.apache.org
>> *Subject: *Re: Accessing Kerberos Secured HDFS Resources from Spark on
>> Mesos
>>
>>   On Fri, Jun 26, 2015 at 3:09 PM, Dave Ariens <dariens@blackberry.com>
>> wrote:
>>
>>>  Would there be any way to have the task instances in the slaves call
>>> the UGI login with a principal/keytab provided to the driver?
>>>
>>
>>  That would only work with a very small number of executors. If you have
>> many login requests in a short period of time with the same principal, the
>> KDC will start to deny logins. That's why delegation tokens are used
>> instead of explicit logins.
>>
>>  --
>> Marcelo
>>
>
>
>
> --
> Marcelo
>

Mime
View raw message