spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ron Gonzalez (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-23527) Error with spark-submit and kerberos with TLS-enabled Hadoop cluster
Date Mon, 09 Apr 2018 23:54:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-23527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431514#comment-16431514
] 

Ron Gonzalez commented on SPARK-23527:
--------------------------------------

My admin team has resolved it, and now I get this same problem when I wrap org.apache.spark.deploy.SparkSubmit
and invoke it directly using Java. 

Is there some kind of jar files or configuration that I need to add into classpath?

> Error with spark-submit and kerberos with TLS-enabled Hadoop cluster
> --------------------------------------------------------------------
>
>                 Key: SPARK-23527
>                 URL: https://issues.apache.org/jira/browse/SPARK-23527
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Submit
>    Affects Versions: 2.2.1
>         Environment: core-site.xml
> <property>
>     <name>hadoop.security.key.provider.path</name>
>     <value>kms://https@host1.domain.com;host2.domain.com:16000/kms</value>
> </property>
> hdfs-site.xml
> <property>
>     <name>dfs.encryption.key.provider.uri</name>
>     <value>kms://https@host1.domain.com;host2.domain.com:16000/kms</value>
> </property>
>            Reporter: Ron Gonzalez
>            Priority: Critical
>
> For current configuration of our enterprise cluster, I submit using spark-submit:
> ./spark-submit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi
--conf spark.yarn.jars=hdfs:/user/user1/spark/lib/*.jar ../examples/jars/spark-examples_2.11-2.2.1.jar 10
> I am getting the following problem:
>  
> 18/02/27 21:03:48 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 3351181 for
svchdc236d on ha-hdfs:nameservice1
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.UnknownHostException:
host1.domain.com;host2.domain.com
>  at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374)
>  at org.apache.hadoop.crypto.key.kms.KMSClientProvider.getDelegationTokenService(KMSClientProvider.java:825)
>  at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:781)
>  at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
>  at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2046)
>  at org.apache.spark.deploy.yarn.security.HadoopFSCredentialProvider$$anonfun$obtainCredentials$1.apply(HadoopFSCredentialProvider.scala:52)
>  
> If I get rid of the other host for the properties so instead of kms://https@host1.domain.com;host2.domain.com:16000/kms,
I convert it to:
> kms://https@host1.domain.com:16000/kms
> it fails with a different error:
> java.io.IOException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException:
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable
to find valid certification path to requested target
> If I do the same thing using spark 1.6, it works so it seems like a regression...
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message