kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chylek, Artur" <Artur.Chy...@sabre.com>
Subject RE: Kafka upgrade process details
Date Thu, 09 May 2019 10:51:26 GMT
Thanks for prompt response.
I am not sure I understand correctly, but I am still confused why switching inter.broker.protocol.version
in the last step would make the process irreversible.
If we agree that log conversion to a new format is applied when a new value of log.message.format.version
or broker's default one is applied then according to documentation switching back to old version
of broker will make no harm. So either a broker is able to read a newer versions of log and
dismiss parts he doesn’t understand or the conversion itself occurs only at certain combination
of inter.broker.protocol.version and log.message.format.version. Or maybe I am completely
wrong with my assumptions.


-----Original Message-----
From: M. Manna [mailto:manmedia@gmail.com] 
Sent: Thursday, May 09, 2019 12:19 PM
To: Kafka Users <users@kafka.apache.org>
Subject: Re: Kafka upgrade process details

Sabre Email Notification: This email is from an EXTERNAL source. Please use caution when clicking
on links or opening attachments from an unknown or suspicious sender.To report a suspected
phishing email, browse to: help.sabre.com > Risk & Security > Report a Security


The upgrade process is such that

1) You ensure that there is a hard-check on protocol version if not exists

already. As you have already mentioned above, in #3 - it's to ensure that

min verson for msg formats are being adhered to before upgrade.

2) broker protocol version is to ensure that when you do rolling upgrade

there is a minimum compatibility between inter-broker comms (if that makes


API versions are there for a specific reason (to ensure compatbility is

maintained first before the protocol/msg formats are fixed). Also, it

ensures that a regular upgrade (e.g. from bugfix->major release) happens

seamlessly via rolling restart. If you want to get techincal about it, you

can look at kafka Table load and config load in github codebase.

Once you've initiated a rolling restart by setting

inter.broker.protocl.version and log.messge.format.version there is no way

back - but i am happy to be proven wrong as I have only done rolling

restart successfully (never needed to go back :) )

On Thu, 9 May 2019 at 09:54, Chylek, Artur <Artur.Chylek@sabre.com> wrote:

> Hi,

> I read the documentation about upgrading Kafka(

> https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.apache.org_21_documentation.html-23upgrade-5F2-5F1-5F0&d=DwIBaQ&c=FXJfUb1oWgygD0uNz-ujnA&r=bVI5M04li2b_AW9E6XWAZb5H4NuzOzdzPeKTA_sjdMg&m=3g5HRSTfPHijn_jhhzYIB88jScofj68jB1AmrcMPBC0&s=qiYRoeea_IxCjRGx1Vi7ylwk2vuCiYGGoPhhGYmiwos&e=)
but I have

> questions that I believe the documentation doesn't cover. I am planning to

> upgrade Kafka from 2.0.0 to 2.1.0 and would like to make sure what to do

> when something goes wrong - I am mostly interested in reverting back to the

> older version. I admit that I am confused about API versions, log message

> versions and inter broker protocol versions.


> According to the documentation these are the steps I need to follow:


>   1.  Set inter.broker.protocol.version to 2.0.0

>   2.  I don't have currently log.message.format.version set to 2.0.0, so

> according to documentation I don't need to specify it

>   3.  Upgrade broker code and restart it. This is I am not sure what

> happens next. Since I don't have log.message.format.version set then

> broker's default value will be used - I guess 2.1.1. Does it mean that:

>      *   New messages that arrive from producers will be saved in a log

> file with a new format?

>      *   Old messages that already exist in log files will be converted to

> the new format once the broker starts?

>      *   Does inter.broker.protocol.version has anything to do with

> log.message.format.version?

> I guess a) and b) are not true since according to the documentation up to

> this point I can revert back to the older version of Kafka. Could you

> confirm?


>   1.  Change inter.broker.protocol.version to 2.1.1 and restart brokers -

> after this is done reverting back is not possible. Could you explain what

> is happening in Kafka so that reverting cannot be done?


> Regards,

> Artur


View raw message