flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8181) Kafka producer's FlinkFixedPartitioner returns different partitions when a target topic is rescaled
Date Fri, 01 Dec 2017 09:23:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274156#comment-16274156

ASF GitHub Bot commented on FLINK-8181:

Github user tzulitai commented on a diff in the pull request:

    --- Diff: flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/partitioner/FlinkFixedPartitioner.java
    @@ -68,6 +78,13 @@ public int partition(T record, byte[] key, byte[] value, String targetTopic,
     			partitions != null && partitions.length > 0,
     			"Partitions of the target topic is empty.");
    -		return partitions[parallelInstanceId % partitions.length];
    +		if (topicToFixedPartition.containsKey(targetTopic)) {
    --- End diff --
    On second thought, passing in the runtime context wouldn’t work. To let user custom
partitioners  be stateful, we’ll essentially need to make FlinkKafkaPartitioner a CheckpointedFunction,
and let the FlinkKafkaProducerBase invoke the checkpoint methods.
    We should be able to avoid breaking the user API by having empty base implementations
for the checkpoint methods on the FlinkKafkaPartitioner.

> Kafka producer's FlinkFixedPartitioner returns different partitions when a target topic
is rescaled
> ---------------------------------------------------------------------------------------------------
>                 Key: FLINK-8181
>                 URL: https://issues.apache.org/jira/browse/FLINK-8181
>             Project: Flink
>          Issue Type: Bug
>          Components: Kafka Connector
>    Affects Versions: 1.4.0, 1.5.0
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>            Priority: Blocker
>             Fix For: 1.4.0, 1.5.0
> On fixing FLINK-6288 and migrating the original {{FlinkFixedPartitioner}} to the new
partitioning API (commit 9ed9b68397b51bfd2b0f6e532212a82f771641bd), the {{FlinkFixedPartitioner}}
no longer returns identical target partitions once a target topic is rescaled.
> This results in a behavioral regression when the {{FlinkFixedPartitioner}} is used.
> The {{FlinkFixedPartitionerTest}} should also be strengthened to cover the target topic
rescaling case.

This message was sent by Atlassian JIRA

View raw message