kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: owner info in zk is not correct
Date Mon, 12 May 2014 15:22:55 GMT
Hello Yonghui,

In 0.7 the consumer rebalance logic is distributed and in some corner cases
such as soft-failure-caused-consecutive rebalances some consumer may
consider the rebalance as complete while others are still trying the
rebalance process. You can check the GC logs on your consumer to verify if
that is the case:

https://issues.apache.org/jira/browse/KAFKA-242

If you bounce the consumers to trigger another rebalance, this issue would
likely to be resolved.

To solve this issue in 0.9 we are moving the group management like load
rebalance from the ZK-based distributed logic into a centralized
coordiantor. Details can be found here:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design

Guozhang


On Mon, May 12, 2014 at 12:48 AM, Yonghui Zhao <zhaoyonghui@gmail.com>wrote:

> Hi,
>
> We are using kafka 0.7.
>
> 2 brokers, each broker has 10 partitions for one topic
> 3 consumers in one consumer group, each consumer create 10 streams.
>
>
> Today, when we want to rollout new service.
> After we restart one consumer we find exceptions and warning.
>
> kafka.common.ConsumerRebalanceFailedException:
> RecommendEvent_sd-sns-relation01.bj-1399630465426-53d3aefc can't rebalance
> after 4 retries
>
>
> [INFO  2014-05-12 15:17:47.364]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [conflict in /consumers/RecommendEvent/owners/sensei/1-2 data:
> RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-2 stored data:
> RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> [INFO  2014-05-12 15:17:47.366]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for the
> partition ownership to be deleted: 1-2]
> [INFO  2014-05-12 15:17:47.375]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [conflict in /consumers/RecommendEvent/owners/sensei/1-3 data:
> RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-3 stored data:
> RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> [INFO  2014-05-12 15:17:47.375]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for the
> partition ownership to be deleted: 1-3]
> [INFO  2014-05-12 15:17:47.385]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [conflict in /consumers/RecommendEvent/owners/sensei/1-5 data:
> RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-5 stored data:
> RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2]
> [INFO  2014-05-12 15:17:47.386]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for the
> partition ownership to be deleted: 1-5]
>
>
>
> And I opened zk viewer.
>
> In zk, we found 2 consumers in ConsumerGroup/ids:
>
> RecommendEvent_sd-sns-relation02.bj-1399635256619-5d8123c6
> RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3
>
>
> And in owners/topic/ we found all partitions are assigned to
> sd-sns-relation03.bj:
>
> Here is the owner info:
> 1:0  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> 1:1  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> 1:2  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> 1:3  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> 1:4  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> 1:5  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> 1:6  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> 1:7  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> 1:8  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> 1:9  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
>
> 2:0  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> 2:1  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> 2:2  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> 2:3  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> 2:4  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> 2:5  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-5
> 2:6  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-6
> 2:7  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-7
> 2:8  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-8
> 2:9  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-9
>
>
> So all partitions are assigned to sd-sns-relation03.bj,  but from logs and
> counter, we are sure sd-sns-relation02.bj has input too.
>
>
> My question is:
>
> 1. why rebalance failed?
> 2. why owner info is wrong?  btw: zkclient is 0.2
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message