I agree with Wenchen that we'd remove the prefix when passing to a source, so you could use the same "spark.yarn.keytab" option in both places. But I think the problem is that "spark.yarn.keytab" still needs to be set, and it clearly isn't in a shared namespace for catalog options. So I think we would still need a solution for existing options. I'm more comfortable with a white list for existing options that we want to maintain compatibility with.


On Mon, Sep 24, 2018 at 11:52 PM tigerquoll <tigerquoll@outlook.com> wrote:
To give some Kerberos specific examples, The spark-submit args:
-–conf spark.yarn.keytab=path_to_keytab -–conf

are currently not passed through to the data sources.

Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Ryan Blue
Software Engineer