spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ajay Chander <itsche...@gmail.com>
Subject Re: Access_Remote_Kerberized_Cluster_Through_Spark
Date Thu, 10 Nov 2016 01:44:22 GMT
Hi Everyone,

I am still trying to figure this one out. I am stuck with this error
"java.io.IOException:
Can't get Master Kerberos principal for use as renewer ". Below is my code.
Can any of you please provide any insights on this? Thanks for your time.


import java.io.{BufferedInputStream, File, FileInputStream}
import java.net.URI

import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.hadoop.io.IOUtils
import org.apache.hadoop.security.UserGroupInformation
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.{SparkConf, SparkContext}


object SparkHdfs {

  def main(args: Array[String]): Unit = {

    System.setProperty("java.security.krb5.conf", new
File("src\\main\\files\\krb5.conf").getAbsolutePath )
    System.setProperty("sun.security.krb5.debug", "true")

    val sparkConf = new SparkConf().setAppName("SparkHdfs").setMaster("local")
    val sc = new SparkContext(sparkConf)
//    Loading remote cluster configurations
    sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\core-site.xml").getAbsolutePath )
    sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\hdfs-site.xml").getAbsolutePath )
    sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\mapred-site.xml").getAbsolutePath )
    sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\yarn-site.xml").getAbsolutePath )
    sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\ssl-client.xml").getAbsolutePath )
    sc.hadoopConfiguration.addResource(new
File("src\\main\\files\\topology.map").getAbsolutePath )

    val conf = new Configuration()
//    Loading remote cluster configurations
    conf.addResource(new Path(new
File("src\\main\\files\\core-site.xml").getAbsolutePath ))
    conf.addResource(new Path(new
File("src\\main\\files\\hdfs-site.xml").getAbsolutePath ))
    conf.addResource(new Path(new
File("src\\main\\files\\mapred-site.xml").getAbsolutePath ))
    conf.addResource(new Path(new
File("src\\main\\files\\yarn-site.xml").getAbsolutePath ))
    conf.addResource(new Path(new
File("src\\main\\files\\ssl-client.xml").getAbsolutePath ))
    conf.addResource(new Path(new
File("src\\main\\files\\topology.map").getAbsolutePath ))

    conf.set("hadoop.security.authentication", "Kerberos")

    UserGroupInformation.setConfiguration(conf)

    UserGroupInformation.loginUserFromKeytab("myusr@INTERNAL.COMPANY.COM",
      new File("src\\main\\files\\myusr.keytab").getAbsolutePath )

    //   SparkHadoopUtil.get.loginUserFromKeytab("tsadusr@INTERNAL.IMSGLOBAL.COM",
    //      new File("src\\main\\files\\tsadusr.keytab").getAbsolutePath)
    //    Getting this error: java.io.IOException: Can't get Master
Kerberos principal for use as renewer

    sc.textFile("hdfs://vm1.comp.com:8020/user/myusr/temp/file1").collect().foreach(println)
    //    Getting this error: java.io.IOException: Can't get Master
Kerberos principal for use as renewer

  }
}




On Mon, Nov 7, 2016 at 9:42 PM, Ajay Chander <itschevva@gmail.com> wrote:

> Did anyone use https://www.codatlas.com/github.com/apache/spark/HEAD/
> core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala to
> interact with secured Hadoop from Spark ?
>
> Thanks,
> Ajay
>
> On Mon, Nov 7, 2016 at 4:37 PM, Ajay Chander <itschevva@gmail.com> wrote:
>
>>
>> Hi Everyone,
>>
>> I am trying to develop a simple codebase on my machine to read data from
>> secured Hadoop cluster. We have a development cluster which is secured
>> through Kerberos and I want to run a Spark job from my IntelliJ to read
>> some sample data from the cluster. Has anyone done this before ? Can you
>> point me to some sample examples?
>>
>> I understand that, if we want to talk to secured cluster, we need to have
>> keytab and principle. I tried using it through
>> UserGroupInformation.loginUserFromKeytab and
>> SparkHadoopUtil.get.loginUserFromKeytab but so far no luck.
>>
>> I have been trying to do this from quite a while ago. Please let me know
>> if you need more info. Thanks
>>
>> Regards,
>> Ajay
>>
>
>

Mime
View raw message