flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tzulitai <...@git.apache.org>
Subject [GitHub] flink pull request #4344: (release-1.3) [FLINK-7195] [kafka] Remove partitio...
Date Fri, 14 Jul 2017 12:03:55 GMT
GitHub user tzulitai opened a pull request:


    (release-1.3) [FLINK-7195] [kafka] Remove partition list querying when restoring state
in FlinkKafkaConsumer

    This issue is a re-appearance of FLINK-6006. On restore, we should not respect any fetched
partitions list from Kafka and perform any filtering of the restored partition states. There
are corner cases where, due to Kafka broker downtime, some partitions may be missing in the
fetched partition list. Therefore, we should not respect the fetched partitions list on restore
time to manipulate the restored state, which may lead to broken state. To be more precise,
we actually should not require fetching partitions on restore.
    We've stepped on our own foot again and reintroduced this bug in ed68fedbe90db03823d75a020510ad3c344fa73e.
This PR adds proper unit tests for this that does not rely on the internal implementations
and test only on public abstractions of `FlinkKafkaConsumerBase`.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tzulitai/flink FLINK-7195

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4344
commit 12af5d8b0e43b62935dc619258fb8f957b11d0bc
Author: Tzu-Li (Gordon) Tai <tzulitai@apache.org>
Date:   2017-07-14T11:51:03Z

    [FLINK-7195] [kafka] Remove partition list querying when restoring state in FlinkKafkaConsumer


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message