spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Casey <>
Subject Re: Kerberos and YARN - functions in spark-shell and spark submit local but not cluster mode
Date Thu, 08 Dec 2016 07:39:56 GMT
I just read an interesting comment on cloudera:

What does it mean by “when the job is submitted,and you have a kinit, you will have TOKEN
to access HDFS, you would need to pass that on, or the KERBEROS ticket” ?

Reference <>
and full quote:

In a cluster which is kerberised there is no SIMPLE authentication. Make sure that you have
run kinit before you run the application.
Second thing to check: In your application you need to do the right thing and either pass
on the TOKEN or a KERBEROS ticket.
When the job is submitted, and you have done a kinit, you will have TOKEN to access HDFS you
would need to pass that on, or the KERBEROS ticket.
You will need to handle this in your code. I can not see exactly what you are doing at that
point in the startup of your code but any HDFS access will require a TOKEN or KERBEROS ticket.

> On 8 Dec 2016, at 08:35, Gerard Casey <> wrote:
> Thanks Marcelo.
> I’ve completely removed it. Ok - even if I read/write from HDFS?
> Trying to the SparkPi example now
> G
>> On 7 Dec 2016, at 22:10, Marcelo Vanzin < <>>
>> Have you removed all the code dealing with Kerberos that you posted?
>> You should not be setting those principal / keytab configs.
>> Literally all you have to do is login with kinit then run spark-submit.
>> Try with the SparkPi example for instance, instead of your own code.
>> If that doesn't work, you have a configuration issue somewhere.
>> On Wed, Dec 7, 2016 at 1:09 PM, Gerard Casey < <>>
>>> Thanks.
>>> I’ve checked the TGT, principal and key tab. Where to next?!
>>>> On 7 Dec 2016, at 22:03, Marcelo Vanzin < <>>
>>>> On Wed, Dec 7, 2016 at 12:15 PM, Gerard Casey <
<>> wrote:
>>>>> Can anyone point me to a tutorial or a run through of how to use Spark
>>>>> Kerberos? This is proving to be quite confusing. Most search results
on the
>>>>> topic point to what needs inputted at the point of `sparks submit` and
>>>>> the changes needed in the actual src/main/.scala file
>>>> You don't need to write any special code to run Spark with Kerberos.
>>>> Just write your application normally, and make sure you're logged in
>>>> to the KDC (i.e. "klist" shows a valid TGT) before running your app.
>>>> --
>>>> Marcelo
>> -- 
>> Marcelo

View raw message