flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aljoscha Krettek (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2922) Add Queryable Window Operator
Date Mon, 26 Oct 2015 16:13:27 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14974474#comment-14974474

Aljoscha Krettek commented on FLINK-2922:

The idea is, for example, that the regular emission of you window result gets stored as final
truth in your database that serves some statistics to users. If you only have this, you always
get your data with a lag of 1 hour.

You could also want to allow users to query the current count inside that 1 hour window. To
do that you need to have a way to match the query to the result. For that, my idea is to have
(conceptually) two output streams. One for the regular window results and another one for
query results. In the query result stream you basically get a tuple (query, window-result).
So that the user can match elements in the query result stream to the queries that they sent.

> Add Queryable Window Operator
> -----------------------------
>                 Key: FLINK-2922
>                 URL: https://issues.apache.org/jira/browse/FLINK-2922
>             Project: Flink
>          Issue Type: Improvement
>          Components: Streaming
>            Reporter: Aljoscha Krettek
>            Assignee: Aljoscha Krettek
> The idea is to provide a window operator that allows to query the current window result
at any time without discarding the current result.
> For example, a user might have an aggregation window operation with tumbling windows
of 1 hour. Now, at any time they might be interested in the current aggregated value for the
currently in-flight hour window.
> The idea is to make the operator a two input operator where normal elements arrive on
input one while queries arrive on input two. The query stream must be keyed by the same key
as the input stream. If an input arrives for a key the current value for that key is emitted
along with the query element so that the user can map the result to the query.

This message was sent by Atlassian JIRA

View raw message