flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei-Che Wei (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLINK-14302) FlinkKafkaInternalProducer should not send `ADD_PARTITIONS_TO_TXN` request if `newPartitionsInTransaction` is empty when enable EoS
Date Wed, 09 Oct 2019 01:53:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947309#comment-16947309
] 

Wei-Che Wei edited comment on FLINK-14302 at 10/9/19 1:52 AM:
--------------------------------------------------------------

Hi [~becket_qin] 

I think the error was from `txnManager.getTransactionState(transactionalId)` [1].

You are right. Maybe we can report this issue back to kafka community as well. For now, could
we conclude the temporary solution that we should prevent from committing empty transaction?
I have opened a pull request already. Could you help me review it and give me advice? I had
a problem about how to add an integration test on it. Thanks.

[1] [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala#L239]


was (Author: tonywei):
Hi [~becket_qin] 

I think the error was from `txnManager.getTransactionState(transactionalId)`.

You are right. Maybe we can report this issue back to kafka community as well. For now, could
we conclude the temporary solution that we should prevent from committing empty transaction?
I have opened a pull request already. Could you help me review it and give me advice? I had
a problem about how to add an integration test on it. Thanks.

[1] [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala#L239]

> FlinkKafkaInternalProducer should not send `ADD_PARTITIONS_TO_TXN` request if `newPartitionsInTransaction`
is empty when enable EoS
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-14302
>                 URL: https://issues.apache.org/jira/browse/FLINK-14302
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / Kafka
>            Reporter: Wei-Che Wei
>            Assignee: Wei-Che Wei
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> As the survey in this mailing list thread [1], kafka server will bind the error with
topic-partition list when it handles `AddPartitionToTxnRequest`. So when the request body
contains no topic-partition, the error won't be sent back to kafka producer client. Moreover,
it producer internal api, it always check if `newPartitionsInTransaction` is empty before
sending ADD_PARTITIONS_TO_TXN request to kafka cluster. We should apply it as well if you
need to explicitly call it in the first commit phase of two-phase commit sink.
> [1] [http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-producer-failed-with-InvalidTxnStateException-when-performing-commit-transaction-td29384.html]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message