kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Yost <hokiege...@gmail.com>
Subject Upgraded brokers from 0.9.0.1 -> 0.10.0.1: how to upgrade message format to 0.10.0.1?
Date Tue, 19 Sep 2017 13:41:46 GMT
Hi Everyone,

We recently upgraded our cluster from 0.9.0.1 to 0.10.0.1 but had to keep
our Kafka clients at 0.9.0.1. We now want to upgrade our clients and,
concurrently, the message version to 0.10.0.1.

When we did the 0.9.0.1 -> 0.10.0.1 broker upgrade we were not able to
upgrade the kafka clients to 0.10.0.1, and we did not set the message
format. As a result the 0.9.0.1 -> 0.10.0.1 format conversion for both
incoming out outgoing messages caused our memory heap requirements to go
from 6 GB up to 12 GB, resulting in long GC pauses that caused our brokers
to crash.

Once I explicitly set the message format to 0.9.0.1 everything was fine.
However, now that I want to convert to the 0.10.0.1 message format, I am
looking for guidance as to the best way to do this. I can switch our Kafka
clients to the 0.10.0.1 release, no problem.

But...our challenge is that we occasionally have to replay topics, and I am
concerned that we will be back to the issue of converting outgoing messages
from 0.9.0.1 to 0.10.0.1 in these replay scenarios as well as in
inter-broker replication.

Please confirm the best way to upgrade our clients and message format to
0.10.0.1 while preserving the stability of our cluster as well as the
ability to replay topics that will have both 0.9.0.1 and 0.10.0.1-formatted
messages.

Thanks

--John

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message