flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From fhueske <...@git.apache.org>
Subject [GitHub] flink issue #4183: [FLINK-6969][table]Add support for deferred computation f...
Date Wed, 28 Jun 2017 14:47:32 GMT
Github user fhueske commented on the issue:

    Hi @sunjincheng121, that's very good input for the discussion!
    1. I would like to avoid to unnecessarily change the API and to deprecate methods and
parameters. IMO, `firstResultTimeOffset` is a better name because it describes that there
is an offset in time to compute the first result. Also, configuring early firing and deferred
computation would be possible (from an API point of view) if we have a dedicated parameter
for each feature. So we would need to check if both parameters are set and throw an exception.
If we only have a single parameter `firstResultTimeOffset` there can be no invalid configuration
because you either compute an early or deferred result.
    2.1. Yes, it is possible to emit records with correct (i.e., window end) timestamps and
this is what we should do.
    2.2. I think we cannot rely on the fact that other operators support deferred computation.
First, this would force us to implement this for all time-based operators (like over windows
and later joins). Second, if we convert a table with a deferred group window aggregation back
into a DataStream, all records of the stream would be late. I just had a discussion with @aljoscha
about this issue. Holding back watermarks is not really possible in Flink and adjusting watermarks
after a window operator incrementally adds latency. 
    @sunjincheng121 I think you are right about the adjustment of the watermarks. The only
approach that doesn't add more latency than the offset is to subtract the offset from the
watermarks at all sources. We would not need to add any custom triggers and all operators
(including future ones) would immediately support deferred computation.
    What do you think @sunjincheng121 and @wuchong?

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message