hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "slim bouguerra (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-21218) KafkaSerDe doesn't support topics created via Confluent Avro serializer
Date Mon, 11 Feb 2019 21:08:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765379#comment-16765379
] 

slim bouguerra commented on HIVE-21218:
---------------------------------------

[~milan.baran] can you please generate the patch with --no-prefix option ?

{code} git diff master --no-prefix > HIVE-21218.2.patch \{code}

also make sure that your master branch is up to date.

> KafkaSerDe doesn't support topics created via Confluent Avro serializer
> -----------------------------------------------------------------------
>
>                 Key: HIVE-21218
>                 URL: https://issues.apache.org/jira/browse/HIVE-21218
>             Project: Hive
>          Issue Type: Bug
>          Components: kafka integration, Serializers/Deserializers
>    Affects Versions: 3.1.1
>            Reporter: Milan Baran
>            Assignee: Milan Baran
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: HIVE-21218.patch
>
>          Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> According to [Google groups|https://groups.google.com/forum/#!topic/confluent-platform/JYhlXN0u9_A] the
Confluent avro serialzier uses propertiary format for kafka value - <magic_byte 0x00><4
bytes of schema ID><regular avro bytes for object that conforms to schema>. 
> This format does not cause any problem for Confluent kafka deserializer which respect
the format however for hive kafka handler its bit a problem to correctly deserialize kafka
value, because Hive uses custom deserializer from bytes to objects and ignores kafka consumer
ser/deser classes provided via table property.
> It would be nice to support Confluent format with magic byte.
> Also it would be great to support Schema registry as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message