flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tzulitai <...@git.apache.org>
Subject [GitHub] flink issue #3031: [FLINK-4616] Added functionality through which watermarks...
Date Thu, 26 Jan 2017 08:36:05 GMT
Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/3031
  
    A re-clarification about backwards compatibility for state type change:
    Currently, one limitation for compatible applications across savepoint restore is that
you can't change the type of state otherwise state restore will fail, therefore not compatible.
The only work around, is to have another field as the new state with the new type, and somehow
try to "encode" / "decode" the watermark state into / from the original `Tuple2<KafkaTopicPartition,
Long>`. I don't think this is easily possible ...
    
    At the same time, there was recent discussion about letting the window operators also
checkpoint watermarks. So perhaps we might not need to let the Kafka sources checkpoint watermarks
in the end, if the window operators already take care of restoring the previous event time.
    What I'm curious about right now is whether or not in the future, redistributions of Kafka
partition states across source subtasks will work well with the checkpointed watermarks in
the downstream window operators. I don't think there should be a problem.
    
    @aljoscha can you perhaps help clarify this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message