spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Does Kafka dependency jars changed for Spark Structured Streaming 2.2.0?
Date Tue, 12 Sep 2017 01:24:26 GMT
Hi All,

Does Kafka dependency jars changed for Spark Structured Streaming 2.2.0?

kafka-clients-0.10.0.1.jar

spark-streaming-kafka-0-10_2.11-2.2.0.jar


1) Above two are the only Kafka related jars or am I missing something?

2)  What is the difference between the above two jars?

3) If I have the following code. Which jar do I use? It looks like I was
able to compile if any one of the jars above thats why its a bit confusing.


StreamingQuery query = sparkSession.readStream()
        .format("kafka")
        .option("kafka.bootstrap.servers", "localhost:9092")
        .option("subscribe", "hello"))
        .option("startingOffsets", "earliest")
        .load()
        .writeStream()
        .format("console")
        .start();

query.awaitTermination();


Thanks

Mime
View raw message