mrql-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Cabrera (JIRA)" <>
Subject [jira] [Closed] (MRQL-66) Add support for MRQL streaming in Flink streaming mode
Date Tue, 19 Sep 2017 22:31:01 GMT


Alan Cabrera closed MRQL-66.
    Resolution: Unresolved

The MRQL podling has been retired.

> Add support for MRQL streaming in Flink streaming mode
> ------------------------------------------------------
>                 Key: MRQL-66
>                 URL:
>             Project: MRQL
>          Issue Type: New Feature
>          Components: Run-Time/Flink, Streaming
>    Affects Versions: 0.9.6
>            Reporter: Leonidas Fegaras
>            Priority: Critical
> The new extension, MRQL Streaming, works fine with Spark Streaming (see MRQL-63) but
it would be nice if we make it work with Flink Streaming too. It was easy to make it work
with Spark Streaming: Data in one sliding window in a Spark's DStream is viewed as an RDD.
So a DStream can be viewed as a continuous sequence of RDDs. A DStream has a method foreachRDD
that applies a function to each RDD in the stream. So to implement MRQL Streaming, we just
had to use the MRQL Spark evaluator (a function from RDD to RDD) as an argument to foreachRDD.
For Flink Streaming, the implementation will be more complicated. A Flink Streaming DataStream
doesn't provide a hook to a DataSet object. I am guessing that this is because Flink Streaming
is far more general than Spark Streaming (it's not just sliding windows) and because Flink
Streaming needs to do special optimizations. So we need to copy the FlinkEvaluator class into
a new class FlinkStreaming and change all methods to be on DataStream instead of DataSet.
Many DataSet methods have an equivalent in DataStream but some are missing. I have already
provided the input formats for streaming (method FlinkStreaming.stream_source) but we need
to write a stream evaluator for MRQL plans.
> Any volunteer?

This message was sent by Atlassian JIRA

View raw message