spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunita Arvind <sunitarv...@gmail.com>
Subject Re: Adhoc queries on Spark 2.0 with Structured Streaming
Date Fri, 06 May 2016 22:11:07 GMT
Agreed.
Just sharing what I saw,
http://www.slideshare.net/databricks/realtime-spark-from-interactive-queries-to-streaming

http://www.slideshare.net/rxin/the-future-of-realtime-in-spark?next_slideshow=3

It claims to support kafka, files and databases. However, continuous SQL
will be available in 2.1 or later only

regards
Sunita


On Fri, May 6, 2016 at 1:06 PM, Michael Malak <michaelmalak@yahoo.com>
wrote:

> At first glance, it looks like the only streaming data sources available
> out of the box from the github master branch are
> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
>  and
> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala
.
> Out of the Jira epic for Structured Streaming
> https://issues.apache.org/jira/browse/SPARK-8360 it would seem the
> still-open https://issues.apache.org/jira/browse/SPARK-10815 "API design:
> data sources and sinks" is relevant here.
>
> In short, it would seem the code is not there yet to create a Kafka-fed
> Dataframe/Dataset that can be queried with Structured Streaming; or if it
> is, it's not obvious how to write such code.
>
>
> ------------------------------
> *From:* Anthony May <anthonymay@gmail.com>
> *To:* Deepak Sharma <deepakmca05@gmail.com>; Sunita Arvind <
> sunitarvind@gmail.com>
> *Cc:* "user@spark.apache.org" <user@spark.apache.org>
> *Sent:* Friday, May 6, 2016 11:50 AM
> *Subject:* Re: Adhoc queries on Spark 2.0 with Structured Streaming
>
> Yeah, there isn't even a RC yet and no documentation but you can work off
> the code base and test suites:
> https://github.com/apache/spark
> And this might help:
>
> https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/streaming/DataFrameReaderWriterSuite.scala
>
> On Fri, 6 May 2016 at 11:07 Deepak Sharma <deepakmca05@gmail.com> wrote:
>
> Spark 2.0 is yet to come out for public release.
> I am waiting to get hands on it as well.
> Please do let me know if i can download source and build spark2.0 from
> github.
>
> Thanks
> Deepak
>
> On Fri, May 6, 2016 at 9:51 PM, Sunita Arvind <sunitarvind@gmail.com>
> wrote:
>
> Hi All,
>
> We are evaluating a few real time streaming query engines and spark is my
> personal choice. The addition of adhoc queries is what is getting me
> further excited about it, however the talks I have heard so far only
> mention about it but do not provide details. I need to build a prototype to
> ensure it works for our use cases.
>
> Can someone point me to relevant material for this.
>
> regards
> Sunita
>
>
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>
>
>
>

Mime
View raw message