flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-4821) Implement rescalable non-partitioned state for Kinesis Connector
Date Mon, 27 Mar 2017 12:44:41 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943180#comment-15943180

ASF GitHub Bot commented on FLINK-4821:

Github user tzulitai commented on the issue:

    Hi @tony810430!
    Sorry for the long pause on this PR. After some back and forth offline discussions with
others on how exactly we want to proceed with this, we decided to stick with using union state
to cope with the shard discovery on restore problem (at least for 1.3.0). Therefore, we can
finally continue work here :-D
    First of all, to use union state, instead of `ListCheckpointed`, we should use `CheckpointedFunction`
instead. There is a PR for exposing union state to the public API (#3508), but in case that
isn't merged yet within the next few days, I suggest that you don't need to be blocked when
you continue your work on this PR. For now, you can cast the operator state store instance
retrieved through the `FunctionInitializationContext` to `DefaultOperatorStateStore` to use
broadcast state.
    One thing to also note, which is missing in you previous work on this, is that we need
a migration path from the old state access (i.e., via `CheckpointedAsynchronously`) to the
new state (i.e. `CheckpointedFunction`).
    The `FlinkKafkaConsumerBase` class in the Kafka connector provides a very good example
of how to do this. Simply put, in the end, the `FlinkKinesisConsumer` should implement both
`CheckpointedRestoring` and `CheckpointedFunction`, and bridge the old state read from the
legacy `restoreState(...)` method to the new `initializeState(...)` method. The bridge would
simply be a field variable in the consumer class.
    The `FlinkKafkaConsumerBase` also serves as a good example of how to use the `CheckpointedFunction`
if you have questions there.
    Let me know if you have any questions with this, and feel free to ping me any time!

> Implement rescalable non-partitioned state for Kinesis Connector
> ----------------------------------------------------------------
>                 Key: FLINK-4821
>                 URL: https://issues.apache.org/jira/browse/FLINK-4821
>             Project: Flink
>          Issue Type: New Feature
>          Components: Kinesis Connector
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Wei-Che Wei
> FLINK-4379 added the rescalable non-partitioned state feature, along with the implementation
for the Kafka connector.
> The AWS Kinesis connector will benefit from the feature and should implement it too.
This ticket tracks progress for this.

This message was sent by Atlassian JIRA

View raw message