beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "peay (JIRA)" <>
Subject [jira] [Created] (BEAM-1573) KafkaIO does not allow using Kafka serializers and deserializers
Date Tue, 28 Feb 2017 13:58:45 GMT
peay created BEAM-1573:

             Summary: KafkaIO does not allow using Kafka serializers and deserializers
                 Key: BEAM-1573
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-extensions
    Affects Versions: 0.5.0, 0.4.0
            Reporter: peay
            Assignee: Davor Bonaci
            Priority: Minor

KafkaIO does not allow to override the serializer and deserializer settings of the Kafka consumer
and producers it uses internally. Instead, it allows to set a `Coder`, and has a simple Kafka
serializer/deserializer wrapper class that calls the coder.

I appreciate that allowing to use Beam coders is good and consistent with the rest of the
system. However, is there a reason to completely disallow to use custom Kafka serializers

This is a limitation when working with an Avro schema registry for instance, which requires
custom serializers. One can write a `Coder` that wraps a custom Kafka serializer, but that
means two levels of un-necessary wrapping.

In addition, the `Coder` abstraction is not equivalent to Kafka's `Serializer` which gets
the topic name as input. Using a `Coder` wrapper would require duplicating the output topic
setting in the argument to `KafkaIO` and when building the wrapper, which is not elegant and
error prone.

This message was sent by Atlassian JIRA

View raw message