kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yifan Ying <nafan...@gmail.com>
Subject Re: Kafka constant shrinking and expanding after deleting a topic
Date Wed, 06 Apr 2016 18:09:45 GMT
Guozhang, thanks for these links.

Hi Alexis, as Guozhang said, yours seems different from our case. We
deleted a topic but caused shrinking/expanding for other topics.

Yifan

On Tue, Apr 5, 2016 at 10:02 PM, Alexis Midon <alexis.midon@airbnb.com>
wrote:

> I ran into the same issue today. In a production cluster, I noticed the
> "Shrinking ISR for partition" log messages for a topic deleted 2 months
> ago.
> Our staging cluster shows the same messages for all the topics deleted in
> that cluster.
> Both 0.8.2
>
> Yifan, Guozhang, did you find a way to get rid of them?
>
> thanks in advance,
> alexis
>
>
> On Tue, Apr 5, 2016 at 4:16 PM Guozhang Wang <wangguoz@gmail.com> wrote:
>
>> It is possible, there are some discussions about a similar issue in KIP:
>>
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-53+-+Add+custom+policies+for+reconnect+attempts+to+NetworkdClient
>>
>> mailing thread:
>>
>> https://www.mail-archive.com/dev@kafka.apache.org/msg46868.html
>>
>>
>>
>> Guozhang
>>
>> On Tue, Apr 5, 2016 at 2:34 PM, Yifan Ying <nafan628@gmail.com> wrote:
>>
>> > Some updates:
>> >
>> > Yesterday, right after release (producers and consumers reconnected to
>> > Kafka/Zookeeper, but no code change in our producers and consumers), all
>> > under replication issues were resolved automatically and no more high
>> > latency in both Kafka and Zookeeper. But right after today's
>> > release(producers and consumers re-connected again), the under
>> replication
>> > and high latency issue happened again. So the all-at-once reconnecting
>> from
>> > producers and consumers would cause the problem? And all these only
>> > happened since I deleted a deprecated topic in production.
>> >
>> > Yifan
>> >
>> > On Tue, Apr 5, 2016 at 9:04 AM, Guozhang Wang <wangguoz@gmail.com>
>> wrote:
>> >
>> >> These configs are mainly dependent on your publish throughput, since
>> the
>> >> replication throughput is higher bounded by the publish throughput. If
>> the
>> >> publish throughput is not high, then setting a lower threshold values
>> in
>> >> these two configs will cause churns in shrinking / expanding ISRs.
>> >>
>> >> Guozhang
>> >>
>> >> On Mon, Apr 4, 2016 at 11:55 PM, Yifan Ying <nafan628@gmail.com>
>> wrote:
>> >>
>> >>> Thanks for replying, Guozhang. We did increase both settings:
>> >>>
>> >>> replica.lag.max.messages=20000
>> >>>
>> >>> replica.lag.time.max.ms=20000
>> >>>
>> >>>
>> >>> But no sure if these are good enough. And yes, that's a good
>> suggestion
>> >>> to monitor ZK performance.
>> >>>
>> >>>
>> >>> Thanks.
>> >>>
>> >>> On Mon, Apr 4, 2016 at 8:58 PM, Guozhang Wang <wangguoz@gmail.com>
>> >>> wrote:
>> >>>
>> >>>> Hmm, it seems like your broker config "replica.lag.max.messages"
and
>> "
>> >>>> replica.lag.time.max.ms" is mis-configed regarding your replication
>> >>>> traffic, and the deletion of the topic actually makes it below the
>> >>>> threshold. What are the config values for these two? And could you
>> try to
>> >>>> increase these configs and see if that helps?
>> >>>>
>> >>>> In 0.8.2.1 Kafka-consumer-offset-checker.sh access ZK to query the
>> >>>> consumer offsets one-by-one, and hence if your ZK read latency is
>> high it
>> >>>> could take long time. You may want to monitor your ZK cluster
>> performance
>> >>>> to check its read / write latencies.
>> >>>>
>> >>>>
>> >>>> Guozhang
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Mon, Apr 4, 2016 at 10:59 AM, Yifan Ying <nafan628@gmail.com>
>> wrote:
>> >>>>
>> >>>>> Hi Guozhang,
>> >>>>>
>> >>>>> It's 0.8.2.1. So it should be fixed? We also tried to start
from
>> >>>>> scratch by wiping out the data directory on both Kafka and
>> Zookeeper. And
>> >>>>> it's odd that the constant shrinking and expanding happened
after
>> fresh
>> >>>>> restart, and high request latency as well. The brokers are using
>> the same
>> >>>>> config before topic deletion.
>> >>>>>
>> >>>>> Another observation is that, using the
>> >>>>> Kafka-consumer-offset-checker.sh is extremely slow. Any suggestion
>> would be
>> >>>>> appreciated! Thanks.
>> >>>>>
>> >>>>> On Sun, Apr 3, 2016 at 2:29 PM, Guozhang Wang <wangguoz@gmail.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Yifan,
>> >>>>>>
>> >>>>>> Are you on 0.8.0 or 0.8.1/2? There are some issues with
zkVersion
>> >>>>>> checking
>> >>>>>> in 0.8.0 that are fixed in later minor releases of 0.8.
>> >>>>>>
>> >>>>>> Guozhang
>> >>>>>>
>> >>>>>> On Fri, Apr 1, 2016 at 7:46 PM, Yifan Ying <nafan628@gmail.com>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>> > Hi All,
>> >>>>>> >
>> >>>>>> > We deleted a deprecated topic on Kafka cluster(0.8)
and started
>> >>>>>> observing
>> >>>>>> > constant 'Expanding ISR for partition' and 'Shrinking
ISR for
>> >>>>>> partition'
>> >>>>>> > for other topics. As a result we saw a huge number
of under
>> >>>>>> replicated
>> >>>>>> > partitions and very high request latency from Kafka.
And it
>> doesn't
>> >>>>>> seem
>> >>>>>> > able to recover itself.
>> >>>>>> >
>> >>>>>> > Anyone knows what caused this issue and how to resolve
it?
>> >>>>>> >
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> -- Guozhang
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Yifan
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> -- Guozhang
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Yifan
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>> >> --
>> >> -- Guozhang
>> >>
>> >
>> >
>> >
>> > --
>> > Yifan
>> >
>> >
>> >
>>
>>
>> --
>> -- Guozhang
>>
>


-- 
Yifan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message