spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: Akka Stream as the source for Spark Streaming. Please advice...
Date Sat, 12 Nov 2016 14:43:13 GMT
Hi,

Just to add to Cody's answer...the following snippet works fine on master:

spark.readStream
  .format("kafka")
  .option("subscribe", "topic")
  .option("kafka.bootstrap.servers", "localhost:9092")
  .load
  .writeStream
  .format("console")
  .start

Don't forget to add spark-sql-kafka-0-10 module to CLASSPATH as follows:

./bin/spark-shell --packages
org.apache.spark:spark-sql-kafka-0-10_2.11:2.1.0-SNAPSHOT

or libraryDependencies in build.sbt for a standalone Spark app.

See KafkaSourceProvider [1].

[1] https://github.com/apache/spark/blob/master/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Thu, Nov 10, 2016 at 8:46 AM, shyla deshpande
<deshpandeshyla@gmail.com> wrote:
> I am using Spark 2.0.1. I wanted to build a data pipeline using Kafka, Spark
> Streaming and Cassandra using Structured Streaming. But the kafka source
> support for Structured Streaming is not yet available. So now I am trying to
> use Akka Stream as the source to Spark Streaming.
>
> Want to make sure I am heading in the right direction. Please direct me to
> any sample code and reading material for this.
>
> Thanks
>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message