spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hadrien Chicault <chicault.hadr...@gmail.com>
Subject Re: Re: Support SqlStreaming in spark
Date Fri, 15 Jun 2018 07:43:01 GMT
Unsuscribe

2018-06-15 9:20 GMT+02:00 stc <qcsd2011@163.com>:

> The repo you give may solve some of SqlStreaming problems, but not
> friendly enough, user need to learn this new syntax.
>
> --
> Jacky Lee
> Mail:qcsd2011@163.com
>
> At 2018-06-15 11:48:01, "Bowden, Chris" <chris.bowden@microfocus.com>
> wrote:
>
> Not sure if there is a question in here, but if you are hinting that
> structured streaming should support a sql interface, spark has appropriate
> extensibility hooks to make it possible. However, the most powerful
> construct in structured streaming is quite difficult to find a sql
> equivalent for (e.g., flatMapGroupsWithState). This repo could use some
> cleanup but is an example of providing a sql interface to a subset of
> structured streaming's functionality: https://github.
> com/vertica/pstl/blob/master/pstl/src/main/antlr4/org/
> apache/spark/sql/catalyst/parser/pstl/PstlSqlBase.g4.
>
> ------------------------------
> *From:* JackyLee <qcsd2011@163.com>
> *Sent:* Thursday, June 14, 2018 7:06:17 PM
> *To:* dev@spark.apache.org
> *Subject:* Support SqlStreaming in spark
>
> Hello
>
> Nowadays, more and more streaming products begin to support SQL streaming,
> such as KafaSQL, Flink SQL and Storm SQL. To support SQL Streaming can not
> only reduce the threshold of streaming, but also make streaming easier to
> be
> accepted by everyone.
>
> At present, StructStreaming is relatively mature, and the StructStreaming
> is
> based on DataSet API, which make it possibal to  provide a SQL portal for
> structstreaming and run structstreaming in SQL.
>
> To support for SQL Streaming, there are two key points:
> 1, Analysis should be able to parse streaming type SQL.
> 2, Analyzer should be able to map metadata information to the corresponding
> Relation.
>
> Running StructStreaming in SQL can bring some benefits.
> 1, Reduce the entry threshold of StructStreaming and attract users more
> easily.
> 2, Encapsulate the meta information of source or sink into table, maintain
> and manage uniformly, and make users more accessible.
> 3. Metadata permissions management, which is based on hive, can control
> StructStreaming's overall authority management scheme more closely.
>
> We have found some ways to solve this problem. It's a pleasure to discuss
> it
> with you.
>
> Thanks,
>
> Jackey Lee
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Mime
View raw message