kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Yost <hokiege...@gmail.com>
Subject Re: Upgraded brokers from 0.9.0.1 -> 0.10.0.1: how to upgrade message format to 0.10.0.1?
Date Tue, 19 Sep 2017 14:25:12 GMT
Ah, cool, thanks Ismael!

--John

On Tue, Sep 19, 2017 at 10:20 AM, Ismael Juma <ismael@juma.me.uk> wrote:

> 0.10.0.1 consumers understand the older formats. So, the conversion only
> happens when the message format is newer than what the consumer
> understands. For the producer side, the conversion is not particularly
> costly since the data is in the heap and, if you use compression, 0.9.0.x
> would do recompression either way.
>
> Ismael
>
> On Tue, Sep 19, 2017 at 2:41 PM, John Yost <hokiegeek2@gmail.com> wrote:
>
> > Hi Everyone,
> >
> > We recently upgraded our cluster from 0.9.0.1 to 0.10.0.1 but had to keep
> > our Kafka clients at 0.9.0.1. We now want to upgrade our clients and,
> > concurrently, the message version to 0.10.0.1.
> >
> > When we did the 0.9.0.1 -> 0.10.0.1 broker upgrade we were not able to
> > upgrade the kafka clients to 0.10.0.1, and we did not set the message
> > format. As a result the 0.9.0.1 -> 0.10.0.1 format conversion for both
> > incoming out outgoing messages caused our memory heap requirements to go
> > from 6 GB up to 12 GB, resulting in long GC pauses that caused our
> brokers
> > to crash.
> >
> > Once I explicitly set the message format to 0.9.0.1 everything was fine.
> > However, now that I want to convert to the 0.10.0.1 message format, I am
> > looking for guidance as to the best way to do this. I can switch our
> Kafka
> > clients to the 0.10.0.1 release, no problem.
> >
> > But...our challenge is that we occasionally have to replay topics, and I
> am
> > concerned that we will be back to the issue of converting outgoing
> messages
> > from 0.9.0.1 to 0.10.0.1 in these replay scenarios as well as in
> > inter-broker replication.
> >
> > Please confirm the best way to upgrade our clients and message format to
> > 0.10.0.1 while preserving the stability of our cluster as well as the
> > ability to replay topics that will have both 0.9.0.1 and
> 0.10.0.1-formatted
> > messages.
> >
> > Thanks
> >
> > --John
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message