flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hequn Cheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-11139) stream non window join support state ttl
Date Fri, 14 Dec 2018 02:03:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-11139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16720833#comment-16720833
] 

Hequn Cheng commented on FLINK-11139:
-------------------------------------

[~zhaoshijie] Do you mean you have already switched to state ttl on your own flink version?
Could you cherry pick the latest improvement of non-window join to your own version? The commit
is [link|https://github.com/apache/flink/commit/5716e4d9f64f957faeebd28647ccf3229598f0a4](FLINK-10543).
You have to notice that the state will not be compatible.

The oom problem may not benefit from the delete feature compared to the old version, because
the previous non-window join register timers in a fixed interval, i.e., only register a new
timer when the old timer is fired. However, the latest improvement should half the number
of timers since it uses one ValueState to control clean up instead of two, while before, left
and right register timers individually. It worth to give it a try. [~zhaoshijie]. Furthermore,
you can reduce the value of max retention time, the problem can also be alleviated(both in
old and new flink version).

Even the new improvement can half the number of timers for join. It will still OOM if the
number of keys is big. In the long term, we need to adapt our timer logic in flink-table to
state ttl? What do you think? [~fhueske]

> stream non window join support state ttl
> ----------------------------------------
>
>                 Key: FLINK-11139
>                 URL: https://issues.apache.org/jira/browse/FLINK-11139
>             Project: Flink
>          Issue Type: Improvement
>          Components: Table API &amp; SQL
>    Affects Versions: 1.7.0
>            Reporter: zhaoshijie
>            Priority: Major
>
> stream non window join function use timer to delete expired data,it is ok for small
amount of data or short expiration time,but it will be OOM(too many timer)on taskManger
 when there  is a long expiration time and  a large amount of data。In fact, table module
other state function has same problem,I would like to contribute to fix it。



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message