kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kamal Chandraprakash <kamal.chandraprak...@gmail.com>
Subject Re: Kafka upgrade process details
Date Fri, 10 May 2019 13:49:51 GMT
Hi,

In Kafka v2.1.0, the OffsetCommit Request/Response schema version is
changed to v4 for the *__consumer_offsets* topic.
If you upgrade Kafka to v2.1.0 & higher and changed the
inter.broker.protocol version to 2.1, then you cannot revert back
to older versions as it doesn't know how to read the *__consumer_offsets *topic
with the latest offset message schema.

Indeed, if you want to revert back to the older versions, you have to
delete the* __consumer_offsets* topic. Note: Once that
topic is deleted, all your consumer groups will start to read from the
latest/earliest available message as per your
`auto.offset.reset` configuration. Please correct me if I am wrong.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets

Note: The `__consumer_offsets` topic is read by the broker after each
restart to load the last commited offset of the consumer groups.
So, once you change the `inter.broker.protocol.version` to 2.1 the
OffsetCommit schema version changes to v4.

On Thu, May 9, 2019 at 4:21 PM Chylek, Artur <Artur.Chylek@sabre.com> wrote:

> Thanks for prompt response.
> I am not sure I understand correctly, but I am still confused why
> switching inter.broker.protocol.version in the last step would make the
> process irreversible.
> If we agree that log conversion to a new format is applied when a new
> value of log.message.format.version or broker's default one is applied then
> according to documentation switching back to old version of broker will
> make no harm. So either a broker is able to read a newer versions of log
> and dismiss parts he doesn’t understand or the conversion itself occurs
> only at certain combination of inter.broker.protocol.version and
> log.message.format.version. Or maybe I am completely wrong with my
> assumptions.
>
> Regards,
> Artur
>
> -----Original Message-----
> From: M. Manna [mailto:manmedia@gmail.com]
> Sent: Thursday, May 09, 2019 12:19 PM
> To: Kafka Users <users@kafka.apache.org>
> Subject: Re: Kafka upgrade process details
>
> Sabre Email Notification: This email is from an EXTERNAL source. Please
> use caution when clicking on links or opening attachments from an unknown
> or suspicious sender.To report a suspected phishing email, browse to:
> help.sabre.com > Risk & Security > Report a Security Issue
>
> ______________________________________________________________________
> Artur,
>
>
>
> The upgrade process is such that
>
>
>
> 1) You ensure that there is a hard-check on protocol version if not exists
>
> already. As you have already mentioned above, in #3 - it's to ensure that
>
> min verson for msg formats are being adhered to before upgrade.
>
> 2) broker protocol version is to ensure that when you do rolling upgrade
>
> there is a minimum compatibility between inter-broker comms (if that makes
>
> sense).
>
>
>
> API versions are there for a specific reason (to ensure compatbility is
>
> maintained first before the protocol/msg formats are fixed). Also, it
>
> ensures that a regular upgrade (e.g. from bugfix->major release) happens
>
> seamlessly via rolling restart. If you want to get techincal about it, you
>
> can look at kafka Table load and config load in github codebase.
>
>
>
> Once you've initiated a rolling restart by setting
>
> inter.broker.protocl.version and log.messge.format.version there is no way
>
> back - but i am happy to be proven wrong as I have only done rolling
>
> restart successfully (never needed to go back :) )
>
>
>
>
>
>
>
> On Thu, 9 May 2019 at 09:54, Chylek, Artur <Artur.Chylek@sabre.com> wrote:
>
>
>
> > Hi,
>
> > I read the documentation about upgrading Kafka(
>
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.apache.org_21_documentation.html-23upgrade-5F2-5F1-5F0&d=DwIBaQ&c=FXJfUb1oWgygD0uNz-ujnA&r=bVI5M04li2b_AW9E6XWAZb5H4NuzOzdzPeKTA_sjdMg&m=3g5HRSTfPHijn_jhhzYIB88jScofj68jB1AmrcMPBC0&s=qiYRoeea_IxCjRGx1Vi7ylwk2vuCiYGGoPhhGYmiwos&e=)
> but I have
>
> > questions that I believe the documentation doesn't cover. I am planning
> to
>
> > upgrade Kafka from 2.0.0 to 2.1.0 and would like to make sure what to do
>
> > when something goes wrong - I am mostly interested in reverting back to
> the
>
> > older version. I admit that I am confused about API versions, log message
>
> > versions and inter broker protocol versions.
>
> >
>
> > According to the documentation these are the steps I need to follow:
>
> >
>
> >   1.  Set inter.broker.protocol.version to 2.0.0
>
> >   2.  I don't have currently log.message.format.version set to 2.0.0, so
>
> > according to documentation I don't need to specify it
>
> >   3.  Upgrade broker code and restart it. This is I am not sure what
>
> > happens next. Since I don't have log.message.format.version set then
>
> > broker's default value will be used - I guess 2.1.1. Does it mean that:
>
> >      *   New messages that arrive from producers will be saved in a log
>
> > file with a new format?
>
> >      *   Old messages that already exist in log files will be converted
> to
>
> > the new format once the broker starts?
>
> >      *   Does inter.broker.protocol.version has anything to do with
>
> > log.message.format.version?
>
> > I guess a) and b) are not true since according to the documentation up to
>
> > this point I can revert back to the older version of Kafka. Could you
>
> > confirm?
>
> >
>
> >   1.  Change inter.broker.protocol.version to 2.1.1 and restart brokers -
>
> > after this is done reverting back is not possible. Could you explain what
>
> > is happening in Kafka so that reverting cannot be done?
>
> >
>
> > Regards,
>
> > Artur
>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message