spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Burak Yavuz <brk...@gmail.com>
Subject Re: Spark Structured Streaming not connecting to Kafka using kerberos
Date Mon, 16 Oct 2017 17:08:35 GMT
Hi Darshan,

How are you creating your kafka stream? Can you please share the options
you provide?

spark.readStream.format("kafka")
  .option(...) // all these please
  .load()


On Sat, Oct 14, 2017 at 1:55 AM, Darshan Pandya <darshanpandya@gmail.com>
wrote:

> Hello,
>
> I'm using Spark 2.1.0 on CDH 5.8 with kafka 0.10.0.1 + kerberos
>
> I am unable to connect to the kafka broker with the following message
>
>
> 17/10/14 14:29:10 WARN clients.NetworkClient: Bootstrap broker
> 10.197.19.25:9092 disconnected
>
> and is unable to consume any messages.
>
> And am using it as follows
>
> jaas.conf
>
> KafkaClient {
> com.sun.security.auth.module.Krb5LoginModule required
> useKeyTab=true
> keyTab="./gandalf.keytab"
> storeKey=true
> useTicketCache=false
> serviceName="kafka"
> principal="gandalf@DOMAIN.COM";
> };
>
> $SPARK_HOME/bin/spark-submit \
> --master yarn \
> --files jaas.conf,gandalf.keytab \
> --driver-java-options "-Djava.security.auth.login.config=./jaas.conf -Dhdp.version=2.4.2.0-258"
\
> --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf"
\
> --class com.example.ClassName uber-jar-with-deps-and-hive-site.jar
>
> Thanks in advance.
>
>
> --
> Sincerely,
> Darshan
>
>

Mime
View raw message