kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Avro serialization
Date Tue, 20 Aug 2013 16:19:29 GMT
Hello Mark,

The Kafka message format starts with a magic byte indicating what kind of
serialization is used for this message. And if this byte indicates Avro,
you can layout your message as starting with the schemaId and then followed
by message payload. Upon consumption, you can first get the schemaId, query
Avro for the schema given the id, and then use schema to deserialize the


On Tue, Aug 20, 2013 at 7:08 AM, Mark <static.void.dev@gmail.com> wrote:

> Can someone break down how message serialization would work with Avro?
> I've read instead of adding a schema to every single event it would be wise
> to add some sort of fingerprint with each message to identify which schema
> it should used. What I'm having trouble understanding is, how do we read
> the fingerprint without a schema? Don't we need the schema to deserialize?
>  Same question goes for working with Hadoop.. how does the input format
> know which schema to use?
> Thanks

-- Guozhang

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message