beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "peay (JIRA)" <>
Subject [jira] [Commented] (BEAM-1573) KafkaIO does not allow using Kafka serializers and deserializers
Date Tue, 28 Feb 2017 19:24:45 GMT


peay commented on BEAM-1573:

Actually, thinking more about it, just setting the serializer class in the Kafka properties
doesn't work in terms of getting a `Write` with the correct key/value types.

Maybe we can instead have a method along the lines of

    public <ValueT> Write<K, ValueT> withCustomKafkaValueSerializer(Serializer<ValueT>
serializer) {
      Map<String, Object> configUpdates = ImmutableMap.of(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
      Map<String, Object> config = updateKafkaProperties(producerConfig,
          TypedWrite.IGNORED_PRODUCER_PROPERTIES, configUpdates);
      return new Write<K, ValueT>(topic, keyCoder, null, config);

> KafkaIO does not allow using Kafka serializers and deserializers
> ----------------------------------------------------------------
>                 Key: BEAM-1573
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>    Affects Versions: 0.4.0, 0.5.0
>            Reporter: peay
>            Assignee: Davor Bonaci
>            Priority: Minor
> KafkaIO does not allow to override the serializer and deserializer settings of the Kafka
consumer and producers it uses internally. Instead, it allows to set a `Coder`, and has a
simple Kafka serializer/deserializer wrapper class that calls the coder.
> I appreciate that allowing to use Beam coders is good and consistent with the rest of
the system. However, is there a reason to completely disallow to use custom Kafka serializers
> This is a limitation when working with an Avro schema registry for instance, which requires
custom serializers. One can write a `Coder` that wraps a custom Kafka serializer, but that
means two levels of un-necessary wrapping.
> In addition, the `Coder` abstraction is not equivalent to Kafka's `Serializer` which
gets the topic name as input. Using a `Coder` wrapper would require duplicating the output
topic setting in the argument to `KafkaIO` and when building the wrapper, which is not elegant
and error prone.

This message was sent by Atlassian JIRA

View raw message