kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark <static.void....@gmail.com>
Subject Re: Avro serialization
Date Tue, 20 Aug 2013 17:54:52 GMT
So basically prepend the information in the header of each message? Wouldn't this require changes
in all readers/writers?

On Aug 20, 2013, at 9:19 AM, Guozhang Wang <wangguoz@gmail.com> wrote:

> Hello Mark,
> The Kafka message format starts with a magic byte indicating what kind of
> serialization is used for this message. And if this byte indicates Avro,
> you can layout your message as starting with the schemaId and then followed
> by message payload. Upon consumption, you can first get the schemaId, query
> Avro for the schema given the id, and then use schema to deserialize the
> message.
> Guozhang
> On Tue, Aug 20, 2013 at 7:08 AM, Mark <static.void.dev@gmail.com> wrote:
>> Can someone break down how message serialization would work with Avro?
>> I've read instead of adding a schema to every single event it would be wise
>> to add some sort of fingerprint with each message to identify which schema
>> it should used. What I'm having trouble understanding is, how do we read
>> the fingerprint without a schema? Don't we need the schema to deserialize?
>> Same question goes for working with Hadoop.. how does the input format
>> know which schema to use?
>> Thanks
> -- 
> -- Guozhang

View raw message