At first glance, it looks like the only streaming data sources available out of the box from the github master branch are https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala and https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala . Out of the Jira epic for Structured Streaming https://issues.apache.org/jira/browse/SPARK-8360 it would seem the still-open https://issues.apache.org/jira/browse/SPARK-10815 "API design: data sources and sinks" is relevant here.In short, it would seem the code is not there yet to create a Kafka-fed Dataframe/Dataset that can be queried with Structured Streaming; or if it is, it's not obvious how to write such code.
From: Anthony May <firstname.lastname@example.org>
To: Deepak Sharma <email@example.com>; Sunita Arvind <firstname.lastname@example.org>
Cc: "email@example.com" <firstname.lastname@example.org>
Sent: Friday, May 6, 2016 11:50 AM
Subject: Re: Adhoc queries on Spark 2.0 with Structured Streaming
Yeah, there isn't even a RC yet and no documentation but you can work off the code base and test suites:
And this might help:
https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/streaming/DataFrameReaderWriterSuite.scalaOn Fri, 6 May 2016 at 11:07 Deepak Sharma <email@example.com> wrote:DeepakThanksPlease do let me know if i can download source and build spark2.0 from github.Spark 2.0 is yet to come out for public release.I am waiting to get hands on it as well.On Fri, May 6, 2016 at 9:51 PM, Sunita Arvind <firstname.lastname@example.org> wrote:SunitaHi All,regards
We are evaluating a few real time streaming query engines and spark is my personal choice. The addition of adhoc queries is what is getting me further excited about it, however the talks I have heard so far only mention about it but do not provide details. I need to build a prototype to ensure it works for our use cases.
Can someone point me to relevant material for this.