kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajiv Kurian <ra...@signalfx.com>
Subject Re: Any gotchas upgrading to 0.9?
Date Mon, 14 Dec 2015 18:46:33 GMT
Scratch that. On more careful observation I do see this in the logs:

 inter.broker.protocol.version = 0.8.2.X

On Mon, Dec 14, 2015 at 10:25 AM, Rajiv Kurian <rajiv@signalfx.com> wrote:

> I am in the process of updating to 0.9 and had another question.
>
> The docs at http://kafka.apache.org/documentation.html#upgrade say that
> to do a smooth upgrade from 0.8.2.X to 0.9 we can use the
> inter.broker.protocol.version config to control what protocol to use. After
> upgrading how can we tell that we are running the right inter broker
> protocol? Is there any jmx metric I could look at to confirm that I started
> the broker correctly? Or maybe something in the logs? I grepped for
> inter.broker.protocol.version in the logs and didn't find anything.
>
> How about when I change the protocol version back to 0.9.0.0? I have
> followed the steps outlined in the guide on a test environment and my
> broker and client metrics look fine. But it would be great to get some kind
> of confirmation that each step (upgrade to 0.9 with 0.8.2.X protocol and
> then restart with 0.9.0.0 protocol) worked.
>
> Thanks,
> Rajiv
>
> On Tue, Dec 1, 2015 at 11:39 AM, Guozhang Wang <wangguoz@gmail.com> wrote:
>
>> Old clients should work well with the new servers, while vice versa is not
>> generally true mainly because the new consumer client uses a few new
>> request types that only the new brokers can recognize. So the normal
>> upgrade path is server-first, clients later.
>>
>> Filed https://issues.apache.org/jira/browse/KAFKA-2923 to update the
>> upgrade doc to include it.
>>
>> Guozhang
>>
>> On Tue, Dec 1, 2015 at 11:14 AM, Rajiv Kurian <rajiv@signalfx.com> wrote:
>>
>> > Thanks folks. Anywhere I can read about client compatibility i.e. old
>> > clients working with new servers and new clients working with old
>> servers?
>> >
>> > Thanks,
>> > Rajiv
>> >
>> > On Tue, Dec 1, 2015 at 10:54 AM, Jay Kreps <jay@confluent.io> wrote:
>> >
>> > > I think the point is that we should ideally try to cover all these in
>> the
>> > > "upgrade" notes.
>> > >
>> > > -Jay
>> > >
>> > > On Tue, Dec 1, 2015 at 10:37 AM, Aditya Auradkar <
>> aauradkar@linkedin.com
>> > >
>> > > wrote:
>> > >
>> > > > Rajiv,
>> > > >
>> > > > By default, the quota is unlimited until you decide to configure
>> them
>> > > > explicitly.
>> > > > And yes, we did get rid of "replica.lag.max.messages", so that
>> > > > configuration will no longer apply.
>> > > >
>> > > > Aditya
>> > > >
>> > > > On Tue, Dec 1, 2015 at 10:24 AM, Todd Snyder <
>> tsnyder@blackberry.com>
>> > > > wrote:
>> > > >
>> > > > > The quota page is here:
>> > > > > http://kafka.apache.org/documentation.html#design_quotas
>> > > > >
>> > > > > "By default, each unique client-id receives a fixed quota in
>> > bytes/sec
>> > > as
>> > > > > configured by the cluster (quota.producer.default,
>> > > > quota.consumer.default)"
>> > > > >
>> > > > >
>> > > > > I also noticed there's been a change in the replication
>> configuration
>> > > > > while reading:
>> > > > >
>> > > >
>> > >
>> >
>> http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
>> > > > >
>> > > > > It may not break anything, but it may impact how you decide to
>> > > configure
>> > > > > and monitor replication
>> > > > >
>> > > > > "Now there is only one value you need to configure on the server
>> > which
>> > > is
>> > > > > replica.lag.time.max.ms. The interpretation of this has changed
>> to
>> > be
>> > > > the
>> > > > > time for which a replica has been out-of-sync with the leader.
>> Stuck
>> > or
>> > > > > failed replicas are detected the same way as before - if a replica
>> > > fails
>> > > > to
>> > > > > send a fetch request for longer than replica.lag.time.max.ms,
it
>> is
>> > > > > considered dead and is removed from the ISR. The mechanism of
>> > detecting
>> > > > > slow replicas has changed - if a replica starts lagging behind
the
>> > > leader
>> > > > > for longer than replica.lag.time.max.ms, then it is considered
>> too
>> > > slow
>> > > > > and is removed from the ISR. So even if there is a spike in
>> traffic
>> > and
>> > > > > large batches of messages are written on the leader, unless the
>> > replica
>> > > > > consistently remains behind the leader for
>> replica.lag.time.max.ms,
>> > it
>> > > > > will not shuffle in and out of the ISR."
>> > > > >
>> > > > >
>> > > > >
>> > > > > -----Original Message-----
>> > > > > From: Rajiv Kurian [mailto:rajiv@signalfx.com]
>> > > > > Sent: Tuesday, December 01, 2015 11:57
>> > > > > To: users@kafka.apache.org
>> > > > > Subject: Re: Any gotchas upgrading to 0.9?
>> > > > >
>> > > > > Also I remember reading (can't find now) something about default
>> > > traffic
>> > > > > quotas. I'd hope the default quotas are very large (infinite?)
and
>> > not
>> > > > > small so that compatibility is maintained. It would be very
>> > unfortunate
>> > > > if
>> > > > > some of our traffic was throttled because of the upgrade because
>> of
>> > > magic
>> > > > > defaults. For example we have a certain cluster dedicated to
>> serving
>> > a
>> > > > > single important topic and we'd hate for it to be throttled
>> because
>> > of
>> > > > > incorrect defaults.
>> > > > >
>> > > > > Thanks,
>> > > > > Rajiv
>> > > > >
>> > > > > On Tue, Dec 1, 2015 at 8:54 AM, Rajiv Kurian <rajiv@signalfx.com>
>> > > wrote:
>> > > > >
>> > > > > > I saw the upgrade path documentation at
>> > > > > > http://kafka.apache.org/documentation.html and that kind
of
>> > answers
>> > > > (1).
>> > > > > > Not sure if there is anything about client compatibility
though.
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Dec 1, 2015 at 8:51 AM, Rajiv Kurian <
>> rajiv@signalfx.com>
>> > > > wrote:
>> > > > > >
>> > > > > >> I plan to upgrade both the server and clients to 0.9.
Had a few
>> > > > > questions
>> > > > > >> before I went ahead with the upgrade:
>> > > > > >>
>> > > > > >> 1. Do all brokers need to be on 0.9? Currently we are
running
>> > 0.8.2.
>> > > > > We'd
>> > > > > >> ideally like to convert only a few brokers to 0.9 and
only if
>> we
>> > > don't
>> > > > > see
>> > > > > >> problems convert the rest.
>> > > > > >>
>> > > > > >> 2. Is it possible to run Kafka 0.9 clients (specifically
the
>> > > consumer)
>> > > > > >> with Kafka 0.8.2 brokers?
>> > > > > >>
>> > > > > >> Any link to the upgrade path would be really useful.
>> > > > > >>
>> > > > > >> Thanks,
>> > > > > >> Rajiv
>> > > > > >>
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message