flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei-Che Wei (Jira)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-14302) FlinkKafkaInternalProducer should not send `ADD_PARTITIONS_TO_TXN` request if `newPartitionsInTransaction` is empty when enable EoS
Date Mon, 07 Oct 2019 01:01:11 GMT

    [ https://issues.apache.org/jira/browse/FLINK-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16945515#comment-16945515

Wei-Che Wei commented on FLINK-14302:

Hi [~becket_qin] 

The root cause of NOT_COORDINATOR is not from empty  partition list.

It is due to that broker who host the original transaction coordinator was restarted. The
error should be propagate to client to refetch new coordinator, but the error will be bind
with each element of partition list. Since this list was empty, the error got lost in response
message. And that lead to the following commit failed. 

For more information, you can refer to my replys in the mailing thread. Thanks. 

> FlinkKafkaInternalProducer should not send `ADD_PARTITIONS_TO_TXN` request if `newPartitionsInTransaction`
is empty when enable EoS
> -----------------------------------------------------------------------------------------------------------------------------------
>                 Key: FLINK-14302
>                 URL: https://issues.apache.org/jira/browse/FLINK-14302
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / Kafka
>            Reporter: Wei-Che Wei
>            Assignee: Wei-Che Wei
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
> As the survey in this mailing list thread [1], kafka server will bind the error with
topic-partition list when it handles `AddPartitionToTxnRequest`. So when the request body
contains no topic-partition, the error won't be sent back to kafka producer client. Moreover,
it producer internal api, it always check if `newPartitionsInTransaction` is empty before
sending ADD_PARTITIONS_TO_TXN request to kafka cluster. We should apply it as well if you
need to explicitly call it in the first commit phase of two-phase commit sink.
> [1] [http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-producer-failed-with-InvalidTxnStateException-when-performing-commit-transaction-td29384.html]

This message was sent by Atlassian Jira

View raw message