kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: owner info in zk is not correct
Date Fri, 16 May 2014 16:19:28 GMT
Hi Yonghui,

Could you check if consumer2's fetcher thread is still alive? Also we have
an entry in FAQ wiki page about "consumer stopped consuming", current
Apache has some issues with the wiki page, but you may want to check it out
once Apache page resumes.

Guozhang


On Wed, May 14, 2014 at 8:59 PM, Yonghui Zhao <zhaoyonghui@gmail.com> wrote:

> Thanks GuoZhang.
>
> After last accident, we stop all consumers and then restart all consumers
> one by one then it is ok.
>
> 2 brokers,  10 partitions / broker,  3 consumers, each create  10 stream
>
> So  consumer1 consumes 10 partitions,  consumer2 consumers another 10
> partitions,  consumer3 is idle.
>
> Today we find some exceptions in consumer2, after these exceptions
> consumer2 doesn't work, no message is consumed.
>
> But in zk I found the owner ship doesn't change,   consumer1 own 10
> partitions  consumer 2 own another 10 partitions/
>
>  How can we avoid this happen again?
>
>
>   From the log we see the error sequence:
>
>
>
>
> *exception during commitOffsets, Reconnect in multifetch due to socket
> error,  rebalance 2 times,   error in FetcherRunnable'Seems the last error
> in FetcherRunnable is deadly, after this error no message is consumed, but
> zk ownership doesn't release.*Here is all the kafka related log in that
> time
> *:*
>
> [WARN  2014-05-13 16:19:05.020]
> kafka.utils.Logging$class.warn(Logging.scala:79)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
> during commitOffsets]
>     at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
>     at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
> [INFO  2014-05-13 16:19:08.991]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c ZK expired;
> release old broker parition ownership; re-register consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c]
> [INFO  2014-05-13 16:19:08.991]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> registering consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in ZK]
> [WARN  2014-05-13 16:19:09.001]
> kafka.utils.Logging$class.warn(Logging.scala:79)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
> during commitOffsets]
>     at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
>     at
>
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
>     at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
> [INFO  2014-05-13 16:19:09.002]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end registering
> consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in ZK]
> [INFO  2014-05-13 16:19:09.003]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> rebalancing consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> [INFO  2014-05-13 16:19:09.063]
> kafka.utils.Logging$class.info(Logging.scala:69)
> [Reconnect in multifetch due to socket error: ]
>     at kafka.utils.Utils$.read(Utils.scala:538)
>     at
>
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>     at kafka.network.Receive$class.readCompletely(Transmission.scala:55)
>     at
>
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>     at kafka.consumer.SimpleConsumer.getResponse(SimpleConsumer.scala:177)
>     at
> kafka.consumer.SimpleConsumer.liftedTree2$1(SimpleConsumer.scala:117)
>     at kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:115)
>     at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
> [INFO  2014-05-13 16:19:09.068]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> [INFO  2014-05-13 16:19:09.069]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> [INFO  2014-05-13 16:19:09.071]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Cleared all relevant queues for this fetcher]
> [INFO  2014-05-13 16:19:09.071]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.071]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.071]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.072]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Cleared the data chunks in all the consumer message iterators]
> [INFO  2014-05-13 16:19:09.073]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing all
> offsets after clearing the fetcher queues]
> [INFO  2014-05-13 16:19:09.073]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> partition ownership]
> [INFO  2014-05-13 16:19:09.073]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing the
> following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
> 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
> with consumers:
> List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> [INFO  2014-05-13 16:19:09.073]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting to
> claim partition 1-8]
> [INFO  2014-05-13 16:19:09.077]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting to
> claim partition 1-4]
> [INFO  2014-05-13 16:19:09.081]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting to
> claim partition 1-3]
> [INFO  2014-05-13 16:19:09.085]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting to
> claim partition 1-2]
> [INFO  2014-05-13 16:19:09.089]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting to
> claim partition 1-1]
> [INFO  2014-05-13 16:19:09.093]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting to
> claim partition 1-5]
> [INFO  2014-05-13 16:19:09.097]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting to
> claim partition 1-6]
> [INFO  2014-05-13 16:19:09.101]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting to
> claim partition 1-9]
> [INFO  2014-05-13 16:19:09.105]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting to
> claim partition 1-7]
> [INFO  2014-05-13 16:19:09.109]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting to
> claim partition 1-0]
> [INFO  2014-05-13 16:19:09.118]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully
> owned partition 1-2 for topic sensei]
> [INFO  2014-05-13 16:19:09.122]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully
> owned partition 1-3 for topic sensei]
> [INFO  2014-05-13 16:19:09.127]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully
> owned partition 1-5 for topic sensei]
> [INFO  2014-05-13 16:19:09.132]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully
> owned partition 1-0 for topic sensei]
> [INFO  2014-05-13 16:19:09.137]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully
> owned partition 1-9 for topic sensei]
> [INFO  2014-05-13 16:19:09.153]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully
> owned partition 1-6 for topic sensei]
> [INFO  2014-05-13 16:19:09.157]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully
> owned partition 1-7 for topic sensei]
> [INFO  2014-05-13 16:19:09.161]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully
> owned partition 1-1 for topic sensei]
> [INFO  2014-05-13 16:19:09.166]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully
> owned partition 1-4 for topic sensei]
> [INFO  2014-05-13 16:19:09.170]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully
> owned partition 1-8 for topic sensei]
> [INFO  2014-05-13 16:19:09.170]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the
> cache]
> [INFO  2014-05-13 16:19:09.171]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
> 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> 1558487487]
> [INFO  2014-05-13 16:19:09.171]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end rebalancing
> consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> [INFO  2014-05-13 16:19:09.172]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.172]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.172]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617 from
> 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.172]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.172]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.173]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.173]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.173]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.173]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.173]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.188]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> rebalancing consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> [INFO  2014-05-13 16:19:09.240]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> [INFO  2014-05-13 16:19:09.240]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> [INFO  2014-05-13 16:19:09.241]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Cleared all relevant queues for this fetcher]
> [INFO  2014-05-13 16:19:09.241]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.241]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.241]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.241]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Cleared the data chunks in all the consumer message iterators]
> [INFO  2014-05-13 16:19:09.242]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing all
> offsets after clearing the fetcher queues]
> [INFO  2014-05-13 16:19:09.285]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> partition ownership]
> [INFO  2014-05-13 16:19:09.333]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing the
> following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
> 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
> with consumers:
> List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> [INFO  2014-05-13 16:19:09.333]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting to
> claim partition 1-8]
> [INFO  2014-05-13 16:19:09.337]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting to
> claim partition 1-4]
> [INFO  2014-05-13 16:19:09.341]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting to
> claim partition 1-3]
> [INFO  2014-05-13 16:19:09.345]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting to
> claim partition 1-2]
> [INFO  2014-05-13 16:19:09.349]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting to
> claim partition 1-1]
> [INFO  2014-05-13 16:19:09.354]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting to
> claim partition 1-5]
> [INFO  2014-05-13 16:19:09.358]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting to
> claim partition 1-6]
> [INFO  2014-05-13 16:19:09.362]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting to
> claim partition 1-9]
> [INFO  2014-05-13 16:19:09.366]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting to
> claim partition 1-7]
> [INFO  2014-05-13 16:19:09.370]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting to
> claim partition 1-0]
> [INFO  2014-05-13 16:19:09.379]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully
> owned partition 1-2 for topic sensei]
> [INFO  2014-05-13 16:19:09.390]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [/consumers/RecommendEvent/owners/sensei/1-3 exists with value
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 during
> connection loss; this is ok]
> [INFO  2014-05-13 16:19:09.390]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully
> owned partition 1-3 for topic sensei]
> [INFO  2014-05-13 16:19:09.395]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully
> owned partition 1-5 for topic sensei]
> [INFO  2014-05-13 16:19:09.400]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully
> owned partition 1-0 for topic sensei]
> [INFO  2014-05-13 16:19:09.405]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully
> owned partition 1-9 for topic sensei]
> [INFO  2014-05-13 16:19:09.409]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully
> owned partition 1-6 for topic sensei]
> [INFO  2014-05-13 16:19:09.417]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [/consumers/RecommendEvent/owners/sensei/1-7 exists with value
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 during
> connection loss; this is ok]
> [INFO  2014-05-13 16:19:09.417]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully
> owned partition 1-7 for topic sensei]
> [INFO  2014-05-13 16:19:09.422]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully
> owned partition 1-1 for topic sensei]
> [INFO  2014-05-13 16:19:09.430]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [/consumers/RecommendEvent/owners/sensei/1-4 exists with value
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 during
> connection loss; this is ok]
> [INFO  2014-05-13 16:19:09.430]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully
> owned partition 1-4 for topic sensei]
> [INFO  2014-05-13 16:19:09.435]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully
> owned partition 1-8 for topic sensei]
> [INFO  2014-05-13 16:19:09.435]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the
> cache]
> [INFO  2014-05-13 16:19:09.435]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
> 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> 1558487487]
> [INFO  2014-05-13 16:19:09.436]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end rebalancing
> consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> [INFO  2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> rebalancing consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> [INFO  2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617 from
> 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.438]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.439]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.439]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.439]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.492]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> [INFO  2014-05-13 16:19:09.492]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> [INFO  2014-05-13 16:19:09.492]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Cleared all relevant queues for this fetcher]
> [INFO  2014-05-13 16:19:09.492]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.492]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.493]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.493]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.493]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.493]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.493]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.493]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.494]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.494]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Clearing the current data chunk for this consumer iterator]
> [INFO  2014-05-13 16:19:09.494]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [Cleared the data chunks in all the consumer message iterators]
> [INFO  2014-05-13 16:19:09.494]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing all
> offsets after clearing the fetcher queues]
> [INFO  2014-05-13 16:19:09.539]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> partition ownership]
> [INFO  2014-05-13 16:19:09.582]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing the
> following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
> 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
> with consumers:
> List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> [INFO  2014-05-13 16:19:09.582]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting to
> claim partition 1-8]
> [INFO  2014-05-13 16:19:09.586]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting to
> claim partition 1-4]
> [INFO  2014-05-13 16:19:09.590]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting to
> claim partition 1-3]
> [INFO  2014-05-13 16:19:09.594]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting to
> claim partition 1-2]
> [INFO  2014-05-13 16:19:09.598]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting to
> claim partition 1-1]
> [INFO  2014-05-13 16:19:09.602]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting to
> claim partition 1-5]
> [INFO  2014-05-13 16:19:09.606]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting to
> claim partition 1-6]
> [INFO  2014-05-13 16:19:09.610]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting to
> claim partition 1-9]
> [INFO  2014-05-13 16:19:09.614]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting to
> claim partition 1-7]
> [INFO  2014-05-13 16:19:09.618]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting to
> claim partition 1-0]
> [INFO  2014-05-13 16:19:09.633]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully
> owned partition 1-2 for topic sensei]
> [INFO  2014-05-13 16:19:09.638]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully
> owned partition 1-3 for topic sensei]
> [INFO  2014-05-13 16:19:09.646]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully
> owned partition 1-5 for topic sensei]
> [INFO  2014-05-13 16:19:09.656]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully
> owned partition 1-0 for topic sensei]
> [INFO  2014-05-13 16:19:09.660]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully
> owned partition 1-9 for topic sensei]
> [INFO  2014-05-13 16:19:09.665]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully
> owned partition 1-6 for topic sensei]
> [INFO  2014-05-13 16:19:09.669]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully
> owned partition 1-7 for topic sensei]
> [INFO  2014-05-13 16:19:09.674]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully
> owned partition 1-1 for topic sensei]
> [INFO  2014-05-13 16:19:09.679]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully
> owned partition 1-4 for topic sensei]
> [INFO  2014-05-13 16:19:09.684]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully
> owned partition 1-8 for topic sensei]
> [INFO  2014-05-13 16:19:09.684]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the
> cache]
> [INFO  2014-05-13 16:19:09.684]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
> 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> 1558487487]
> [INFO  2014-05-13 16:19:09.684]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end rebalancing
> consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> [INFO  2014-05-13 16:19:09.685]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.685]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.685]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617 from
> 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.685]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.686]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.686]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.686]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.686]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.686]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587
> from 10.101.10.12:9092]
> [INFO  2014-05-13 16:19:09.686]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487
> from 10.101.10.12:9092]
> [ERROR 2014-05-13 16:19:18.687]
> kafka.utils.Logging$class.error(Logging.scala:89) [error in FetcherRunnable
> ]
>     at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:49)
>     at
> kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:186)
>     at kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:113)
>     at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
> [INFO  2014-05-13 16:19:18.687]
> kafka.utils.Logging$class.info(Logging.scala:61)
> [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
>
>
>
> 2014-05-12 23:22 GMT+08:00 Guozhang Wang <wangguoz@gmail.com>:
>
> > Hello Yonghui,
> >
> > In 0.7 the consumer rebalance logic is distributed and in some corner
> cases
> > such as soft-failure-caused-consecutive rebalances some consumer may
> > consider the rebalance as complete while others are still trying the
> > rebalance process. You can check the GC logs on your consumer to verify
> if
> > that is the case:
> >
> > https://issues.apache.org/jira/browse/KAFKA-242
> >
> > If you bounce the consumers to trigger another rebalance, this issue
> would
> > likely to be resolved.
> >
> > To solve this issue in 0.9 we are moving the group management like load
> > rebalance from the ZK-based distributed logic into a centralized
> > coordiantor. Details can be found here:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
> >
> > Guozhang
> >
> >
> > On Mon, May 12, 2014 at 12:48 AM, Yonghui Zhao <zhaoyonghui@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > We are using kafka 0.7.
> > >
> > > 2 brokers, each broker has 10 partitions for one topic
> > > 3 consumers in one consumer group, each consumer create 10 streams.
> > >
> > >
> > > Today, when we want to rollout new service.
> > > After we restart one consumer we find exceptions and warning.
> > >
> > > kafka.common.ConsumerRebalanceFailedException:
> > > RecommendEvent_sd-sns-relation01.bj-1399630465426-53d3aefc can't
> > rebalance
> > > after 4 retries
> > >
> > >
> > > [INFO  2014-05-12 15:17:47.364]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [conflict in /consumers/RecommendEvent/owners/sensei/1-2 data:
> > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-2 stored
> data:
> > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > > [INFO  2014-05-12 15:17:47.366]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for
> > the
> > > partition ownership to be deleted: 1-2]
> > > [INFO  2014-05-12 15:17:47.375]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [conflict in /consumers/RecommendEvent/owners/sensei/1-3 data:
> > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-3 stored
> data:
> > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > > [INFO  2014-05-12 15:17:47.375]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for
> > the
> > > partition ownership to be deleted: 1-3]
> > > [INFO  2014-05-12 15:17:47.385]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [conflict in /consumers/RecommendEvent/owners/sensei/1-5 data:
> > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-5 stored
> data:
> > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2]
> > > [INFO  2014-05-12 15:17:47.386]
> > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for
> > the
> > > partition ownership to be deleted: 1-5]
> > >
> > >
> > >
> > > And I opened zk viewer.
> > >
> > > In zk, we found 2 consumers in ConsumerGroup/ids:
> > >
> > > RecommendEvent_sd-sns-relation02.bj-1399635256619-5d8123c6
> > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3
> > >
> > >
> > > And in owners/topic/ we found all partitions are assigned to
> > > sd-sns-relation03.bj:
> > >
> > > Here is the owner info:
> > > 1:0  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > 1:1  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > 1:2  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > 1:3  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > 1:4  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > 1:5  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > 1:6  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > 1:7  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > 1:8  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > 1:9  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > >
> > > 2:0  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > 2:1  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > 2:2  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > 2:3  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > 2:4  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > 2:5  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-5
> > > 2:6  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-6
> > > 2:7  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-7
> > > 2:8  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-8
> > > 2:9  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-9
> > >
> > >
> > > So all partitions are assigned to sd-sns-relation03.bj,  but from logs
> > and
> > > counter, we are sure sd-sns-relation02.bj has input too.
> > >
> > >
> > > My question is:
> > >
> > > 1. why rebalance failed?
> > > 2. why owner info is wrong?  btw: zkclient is 0.2
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message