flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-1967) Introduce (Event)time in Streaming
Date Thu, 02 Jul 2015 09:25:04 GMT

    [ https://issues.apache.org/jira/browse/FLINK-1967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14611693#comment-14611693

ASF GitHub Bot commented on FLINK-1967:

Github user StephanEwen commented on the pull request:

    Concerning watermarks and performance: Have we considered sending watermarks as records,
rather than events?
      - Events in data channels are inevitably more heavyweight than records. They have less
efficient serialization and flush buffers.
      - In cases where we can guarantee that records have ascending time stamps in a stream
(sub) partition, every record is a watermark by itself, and there is no need to have extra
    The downside of having records rather than events is that we sometimes really want to
flush the buffers with a watermark. On the other hand, buffers are flushed periodically anyways,
so no big deal, probably.
    If our stream records are always 2-tuples (timestamp, record) , then we could do something
like shipping (timestamp, null) records as watermarks.

> Introduce (Event)time in Streaming
> ----------------------------------
>                 Key: FLINK-1967
>                 URL: https://issues.apache.org/jira/browse/FLINK-1967
>             Project: Flink
>          Issue Type: Improvement
>            Reporter: Aljoscha Krettek
>            Assignee: Aljoscha Krettek
> This requires introducing a timestamp in streaming record and a change in the sources
to add timestamps to records. This will also introduce punctuations (or low watermarks) to
allow windows to work correctly on unordered, timestamped input data. In the process of this,
the windowing subsystem also needs to be adapted to use the punctuations. Furthermore, all
operators need to be made aware of punctuations and correctly forward them. Then, a new operator
must be introduced to to allow modification of timestamps.

This message was sent by Atlassian JIRA

View raw message