spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amol Zambare <amol.zamb...@outlook.com>
Subject spark kafka consumer with kerberos - login error
Date Tue, 19 Jun 2018 15:32:03 GMT
I am working on a spark job which reads from kafka topic and write to HDFS however while submitting
the job using spark-submit command I am getting following error.


Error log

 Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException:
Could not login: the client is being asked for a password, but the Kafka client code does
not currently support obtaining a password from the user. Make sure -Djava.security.auth.login.config
property passed to JVM and the client is configured to use a ticket cache (using the JAAS
configuration setting 'useTicketCache=true)'. Make sure you are using FQDN of the Kafka broker
you are trying to connect to. not available to garner  authentication information from the
user


I am passing the user keytab and kafka_client_jaas.conf file to spark submit command as suggested
in Hortonworks documentation or https://github.com/hortonworks-spark/skc#running-on-a-kerberos-enabled-cluster


Passing following parameters to spark-submit

--files user.keytab,kafka_client_jaas.conf \

--driver-java-options "-Djava.security.auth.login.config=kafka_client_jaas.conf" \

--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf"
\

Version information

Spark version - 2.2.0.2.6.4.0-91
Kafka version - 0.10.1

Any help is much appreciated.

Thanks,
Amol


Mime
View raw message