spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-1569) Spark on Yarn, authentication broken by pr299
Date Tue, 29 Apr 2014 19:59:16 GMT

    [ https://issues.apache.org/jira/browse/SPARK-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984713#comment-13984713
] 

Thomas Graves commented on SPARK-1569:
--------------------------------------

<quote>
@tgravescs ah I see, you're right. I think I assumed incorrectly that the executor launcher
would bundle up the options and send them over, but I don't actually see that happening anywhere.
So this part of the code is actually not used:

https://github.com/apache/spark/blob/df6d81425bf3b8830988288069f6863de873aee2/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala#L328

What happens is the executor is just getting its configuration from the driver when the executor
launches. And that works in most cases except for security, which it needs to know about before
connecting. Is that right?
<quote>

That is correct. It needs it before connecting.  The code you reference handles add it for
the application master but not the executors. it looks like we need similar code in ExecutorRunnableUtil.prepareCommand.


Yes before we only had SPARK_JAVA_OPTS and that got put as -D on the command line so it was
always set correctly when the executors launched.


> Spark on Yarn, authentication broken by pr299
> ---------------------------------------------
>
>                 Key: SPARK-1569
>                 URL: https://issues.apache.org/jira/browse/SPARK-1569
>             Project: Spark
>          Issue Type: Sub-task
>          Components: YARN
>    Affects Versions: 1.0.0
>            Reporter: Thomas Graves
>            Priority: Blocker
>
> https://github.com/apache/spark/pull/299 changed the way configuration was done and passed
to the executors.  This breaks use of authentication as the executor needs to know that authentication
is enabled before connecting to the driver.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message