flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7902) TwoPhaseCommitSinkFunctions should use custom TypeSerializer
Date Wed, 01 Nov 2017 09:22:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233826#comment-16233826

ASF GitHub Bot commented on FLINK-7902:

Github user aljoscha commented on the issue:

    Thanks for reviewing, @kl0u! 😃 

> TwoPhaseCommitSinkFunctions should use custom TypeSerializer
> ------------------------------------------------------------
>                 Key: FLINK-7902
>                 URL: https://issues.apache.org/jira/browse/FLINK-7902
>             Project: Flink
>          Issue Type: Bug
>          Components: Kafka Connector
>    Affects Versions: 1.4.0
>            Reporter: Aljoscha Krettek
>            Assignee: Aljoscha Krettek
>            Priority: Blocker
>             Fix For: 1.4.0
> Currently, the {{FlinkKafkaProducer011}} uses {{TypeInformation.of(new TypeHint<State<KafkaTransactionState,
KafkaTransactionContext>>() {})}} to create a {{TypeInformation}} which in turn is used
to create a {{StateDescriptor}} for the state that the Kafka sink stores.
> Behind the scenes, this would be roughly analysed as a {{PojoType(GenericType<KafkaTransactionState>,
GenericType<KafkaTransactionContext>)}} which means we don't have explicit control over
the serialisation format and we also use Kryo (which is the default for {{GenericTypeInfo}}).
This can be problematic if we want to evolve the state schema in the future or if we want
to change Kryo versions.
> We should change {{TwoPhaseCommitSinkFunction}} to only have this constructor:
> {code}
> public TwoPhaseCommitSinkFunction(TypeSerializer<State<TXN, CONTEXT>> stateSerializer)
> {code}
> and we should then change the {{FlinkKafkaProducer011}} to hand in a custom-made {{TypeSerializer}}
for the state.

This message was sent by Atlassian JIRA

View raw message