kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Helin Xiang <xkee...@gmail.com>
Subject Re: owner info in zk is not correct
Date Fri, 23 May 2014 14:49:10 GMT
Thanks Jun.
That's really helpful.


On Fri, May 23, 2014 at 10:38 PM, Jun Rao <junrao@gmail.com> wrote:

> This is true in 0.7.x. In 0.8.x, the fetcher will be restarted
> automatically on such failures.
>
> Thanks,
>
> Jun
>
>
> On Thu, May 22, 2014 at 10:40 PM, Helin Xiang <xkeegan@gmail.com> wrote:
>
> > Hi Guozhang,
> >
> > sorry for asking a little unrelated problem here.
> >
> > We found a consumer stopping fetching data when doing an network upgrade,
> > if the consumer have connection problem with one broker (but OK with
> > zookeeper and other brokers)
> > the fetcherrunnable will stop, but there is no chance to restart the
> thread
> > again ( since there is no zookeeper timeout so no reblance will be
> > triggered)
> >
> > kafka version is 0.7.2 and we are using high level consumer.
> >
> > I did a simulation test using iptables and the result is same( no chance
> to
> > restart fetcherrunnable).
> > I have read the code, seems the exception will not be handled outside.
> Am I
> > wrong about this?
> >
> > THANKS.
> >
> >
> >
> >
> >
> > On Sat, May 17, 2014 at 12:19 AM, Guozhang Wang <wangguoz@gmail.com>
> > wrote:
> >
> > > Hi Yonghui,
> > >
> > > Could you check if consumer2's fetcher thread is still alive? Also we
> > have
> > > an entry in FAQ wiki page about "consumer stopped consuming", current
> > > Apache has some issues with the wiki page, but you may want to check it
> > out
> > > once Apache page resumes.
> > >
> > > Guozhang
> > >
> > >
> > > On Wed, May 14, 2014 at 8:59 PM, Yonghui Zhao <zhaoyonghui@gmail.com>
> > > wrote:
> > >
> > > > Thanks GuoZhang.
> > > >
> > > > After last accident, we stop all consumers and then restart all
> > consumers
> > > > one by one then it is ok.
> > > >
> > > > 2 brokers,  10 partitions / broker,  3 consumers, each create  10
> > stream
> > > >
> > > > So  consumer1 consumes 10 partitions,  consumer2 consumers another 10
> > > > partitions,  consumer3 is idle.
> > > >
> > > > Today we find some exceptions in consumer2, after these exceptions
> > > > consumer2 doesn't work, no message is consumed.
> > > >
> > > > But in zk I found the owner ship doesn't change,   consumer1 own 10
> > > > partitions  consumer 2 own another 10 partitions/
> > > >
> > > >  How can we avoid this happen again?
> > > >
> > > >
> > > >   From the log we see the error sequence:
> > > >
> > > >
> > > >
> > > >
> > > > *exception during commitOffsets, Reconnect in multifetch due to
> socket
> > > > error,  rebalance 2 times,   error in FetcherRunnable'Seems the last
> > > error
> > > > in FetcherRunnable is deadly, after this error no message is
> consumed,
> > > but
> > > > zk ownership doesn't release.*Here is all the kafka related log in
> that
> > > > time
> > > > *:*
> > > >
> > > > [WARN  2014-05-13 16:19:05.020]
> > > > kafka.utils.Logging$class.warn(Logging.scala:79)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
> > > > during commitOffsets]
> > > >     at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
> > > >     at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
> > > > [INFO  2014-05-13 16:19:08.991]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c ZK
> expired;
> > > > release old broker parition ownership; re-register consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c]
> > > > [INFO  2014-05-13 16:19:08.991]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > > > registering consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in ZK]
> > > > [WARN  2014-05-13 16:19:09.001]
> > > > kafka.utils.Logging$class.warn(Logging.scala:79)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
> > > > during commitOffsets]
> > > >     at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
> > > >     at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
> > > > [INFO  2014-05-13 16:19:09.002]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> > > registering
> > > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> in
> > > ZK]
> > > > [INFO  2014-05-13 16:19:09.003]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > > > rebalancing consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> > > > [INFO  2014-05-13 16:19:09.063]
> > > > kafka.utils.Logging$class.info(Logging.scala:69)
> > > > [Reconnect in multifetch due to socket error: ]
> > > >     at kafka.utils.Utils$.read(Utils.scala:538)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> > > >     at
> > kafka.network.Receive$class.readCompletely(Transmission.scala:55)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > > >     at
> > > kafka.consumer.SimpleConsumer.getResponse(SimpleConsumer.scala:177)
> > > >     at
> > > > kafka.consumer.SimpleConsumer.liftedTree2$1(SimpleConsumer.scala:117)
> > > >     at
> > kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:115)
> > > >     at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
> > > > [INFO  2014-05-13 16:19:09.068]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> > > > [INFO  2014-05-13 16:19:09.069]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> > > > [INFO  2014-05-13 16:19:09.071]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Cleared all relevant queues for this fetcher]
> > > > [INFO  2014-05-13 16:19:09.071]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.071]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.071]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.072]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Cleared the data chunks in all the consumer message iterators]
> > > > [INFO  2014-05-13 16:19:09.073]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> Committing
> > > all
> > > > offsets after clearing the fetcher queues]
> > > > [INFO  2014-05-13 16:19:09.073]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> > > > partition ownership]
> > > > [INFO  2014-05-13 16:19:09.073]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> rebalancing
> > > the
> > > > following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7,
> 1-8,
> > > > 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic
> sensei
> > > > with consumers:
> > > > List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> > > > [INFO  2014-05-13 16:19:09.073]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8
> attempting
> > > to
> > > > claim partition 1-8]
> > > > [INFO  2014-05-13 16:19:09.077]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4
> attempting
> > > to
> > > > claim partition 1-4]
> > > > [INFO  2014-05-13 16:19:09.081]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3
> attempting
> > > to
> > > > claim partition 1-3]
> > > > [INFO  2014-05-13 16:19:09.085]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2
> attempting
> > > to
> > > > claim partition 1-2]
> > > > [INFO  2014-05-13 16:19:09.089]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1
> attempting
> > > to
> > > > claim partition 1-1]
> > > > [INFO  2014-05-13 16:19:09.093]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5
> attempting
> > > to
> > > > claim partition 1-5]
> > > > [INFO  2014-05-13 16:19:09.097]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6
> attempting
> > > to
> > > > claim partition 1-6]
> > > > [INFO  2014-05-13 16:19:09.101]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9
> attempting
> > > to
> > > > claim partition 1-9]
> > > > [INFO  2014-05-13 16:19:09.105]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7
> attempting
> > > to
> > > > claim partition 1-7]
> > > > [INFO  2014-05-13 16:19:09.109]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0
> attempting
> > > to
> > > > claim partition 1-0]
> > > > [INFO  2014-05-13 16:19:09.118]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2
> > successfully
> > > > owned partition 1-2 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.122]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3
> > successfully
> > > > owned partition 1-3 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.127]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5
> > successfully
> > > > owned partition 1-5 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.132]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0
> > successfully
> > > > owned partition 1-0 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.137]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9
> > successfully
> > > > owned partition 1-9 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.153]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6
> > successfully
> > > > owned partition 1-6 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.157]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7
> > successfully
> > > > owned partition 1-7 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.161]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1
> > successfully
> > > > owned partition 1-1 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.166]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4
> > successfully
> > > > owned partition 1-4 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.170]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8
> > successfully
> > > > owned partition 1-8 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.170]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating
> > the
> > > > cache]
> > > > [INFO  2014-05-13 16:19:09.171]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> > > > partitions : sensei:1-0: fetched offset = 1555136587: consumed
> offset =
> > > > 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> > > > 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> > > > 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> > > > 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> > > > 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> > > > 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> > > > 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> > > > 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> > > > 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> > > > 1558487487]
> > > > [INFO  2014-05-13 16:19:09.171]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> > > rebalancing
> > > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> try
> > > #0]
> > > > [INFO  2014-05-13 16:19:09.172]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 6 offset:
> > 1563630746
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.172]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 7 offset:
> > 1552261483
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.172]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 4 offset:
> 10316617
> > > from
> > > > 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.172]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 5 offset:
> > 1550214323
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.172]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 2 offset:
> > 1551158177
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.173]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 3 offset:
> > 1552939302
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.173]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 1 offset:
> > 1550960225
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.173]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 8 offset:
> > 1556683052
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.173]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 0 offset:
> > 1555136587
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.173]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 9 offset:
> > 1558487487
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.188]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > > > rebalancing consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> > > > [INFO  2014-05-13 16:19:09.240]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> > > > [INFO  2014-05-13 16:19:09.240]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> > > > [INFO  2014-05-13 16:19:09.241]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Cleared all relevant queues for this fetcher]
> > > > [INFO  2014-05-13 16:19:09.241]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.241]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.241]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.241]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Cleared the data chunks in all the consumer message iterators]
> > > > [INFO  2014-05-13 16:19:09.242]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> Committing
> > > all
> > > > offsets after clearing the fetcher queues]
> > > > [INFO  2014-05-13 16:19:09.285]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> > > > partition ownership]
> > > > [INFO  2014-05-13 16:19:09.333]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> rebalancing
> > > the
> > > > following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7,
> 1-8,
> > > > 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic
> sensei
> > > > with consumers:
> > > > List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> > > > [INFO  2014-05-13 16:19:09.333]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8
> attempting
> > > to
> > > > claim partition 1-8]
> > > > [INFO  2014-05-13 16:19:09.337]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4
> attempting
> > > to
> > > > claim partition 1-4]
> > > > [INFO  2014-05-13 16:19:09.341]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3
> attempting
> > > to
> > > > claim partition 1-3]
> > > > [INFO  2014-05-13 16:19:09.345]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2
> attempting
> > > to
> > > > claim partition 1-2]
> > > > [INFO  2014-05-13 16:19:09.349]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1
> attempting
> > > to
> > > > claim partition 1-1]
> > > > [INFO  2014-05-13 16:19:09.354]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5
> attempting
> > > to
> > > > claim partition 1-5]
> > > > [INFO  2014-05-13 16:19:09.358]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6
> attempting
> > > to
> > > > claim partition 1-6]
> > > > [INFO  2014-05-13 16:19:09.362]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9
> attempting
> > > to
> > > > claim partition 1-9]
> > > > [INFO  2014-05-13 16:19:09.366]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7
> attempting
> > > to
> > > > claim partition 1-7]
> > > > [INFO  2014-05-13 16:19:09.370]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0
> attempting
> > > to
> > > > claim partition 1-0]
> > > > [INFO  2014-05-13 16:19:09.379]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2
> > successfully
> > > > owned partition 1-2 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.390]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [/consumers/RecommendEvent/owners/sensei/1-3 exists with value
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 during
> > > > connection loss; this is ok]
> > > > [INFO  2014-05-13 16:19:09.390]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3
> > successfully
> > > > owned partition 1-3 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.395]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5
> > successfully
> > > > owned partition 1-5 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.400]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0
> > successfully
> > > > owned partition 1-0 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.405]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9
> > successfully
> > > > owned partition 1-9 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.409]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6
> > successfully
> > > > owned partition 1-6 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.417]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [/consumers/RecommendEvent/owners/sensei/1-7 exists with value
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 during
> > > > connection loss; this is ok]
> > > > [INFO  2014-05-13 16:19:09.417]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7
> > successfully
> > > > owned partition 1-7 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.422]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1
> > successfully
> > > > owned partition 1-1 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.430]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [/consumers/RecommendEvent/owners/sensei/1-4 exists with value
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 during
> > > > connection loss; this is ok]
> > > > [INFO  2014-05-13 16:19:09.430]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4
> > successfully
> > > > owned partition 1-4 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.435]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8
> > successfully
> > > > owned partition 1-8 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.435]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating
> > the
> > > > cache]
> > > > [INFO  2014-05-13 16:19:09.435]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> > > > partitions : sensei:1-0: fetched offset = 1555136587: consumed
> offset =
> > > > 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> > > > 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> > > > 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> > > > 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> > > > 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> > > > 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> > > > 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> > > > 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> > > > 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> > > > 1558487487]
> > > > [INFO  2014-05-13 16:19:09.436]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> > > rebalancing
> > > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> try
> > > #0]
> > > > [INFO  2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 6 offset:
> > 1563630746
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
> > > > rebalancing consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
> > > > [INFO  2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 7 offset:
> > 1552261483
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 4 offset:
> 10316617
> > > from
> > > > 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 5 offset:
> > 1550214323
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 2 offset:
> > 1551158177
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 3 offset:
> > 1552939302
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.438]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 1 offset:
> > 1550960225
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.439]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 8 offset:
> > 1556683052
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.439]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 0 offset:
> > 1555136587
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.439]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 9 offset:
> > 1558487487
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.492]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
> > > > [INFO  2014-05-13 16:19:09.492]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> > > > [INFO  2014-05-13 16:19:09.492]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Cleared all relevant queues for this fetcher]
> > > > [INFO  2014-05-13 16:19:09.492]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.492]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.493]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.493]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.493]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.493]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.493]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.493]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.494]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.494]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Clearing the current data chunk for this consumer iterator]
> > > > [INFO  2014-05-13 16:19:09.494]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [Cleared the data chunks in all the consumer message iterators]
> > > > [INFO  2014-05-13 16:19:09.494]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> Committing
> > > all
> > > > offsets after clearing the fetcher queues]
> > > > [INFO  2014-05-13 16:19:09.539]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
> > > > partition ownership]
> > > > [INFO  2014-05-13 16:19:09.582]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> rebalancing
> > > the
> > > > following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7,
> 1-8,
> > > > 1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic
> sensei
> > > > with consumers:
> > > > List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
> > > > RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
> > > > RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
> > > > [INFO  2014-05-13 16:19:09.582]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8
> attempting
> > > to
> > > > claim partition 1-8]
> > > > [INFO  2014-05-13 16:19:09.586]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4
> attempting
> > > to
> > > > claim partition 1-4]
> > > > [INFO  2014-05-13 16:19:09.590]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3
> attempting
> > > to
> > > > claim partition 1-3]
> > > > [INFO  2014-05-13 16:19:09.594]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2
> attempting
> > > to
> > > > claim partition 1-2]
> > > > [INFO  2014-05-13 16:19:09.598]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1
> attempting
> > > to
> > > > claim partition 1-1]
> > > > [INFO  2014-05-13 16:19:09.602]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5
> attempting
> > > to
> > > > claim partition 1-5]
> > > > [INFO  2014-05-13 16:19:09.606]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6
> attempting
> > > to
> > > > claim partition 1-6]
> > > > [INFO  2014-05-13 16:19:09.610]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9
> attempting
> > > to
> > > > claim partition 1-9]
> > > > [INFO  2014-05-13 16:19:09.614]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7
> attempting
> > > to
> > > > claim partition 1-7]
> > > > [INFO  2014-05-13 16:19:09.618]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0
> attempting
> > > to
> > > > claim partition 1-0]
> > > > [INFO  2014-05-13 16:19:09.633]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2
> > successfully
> > > > owned partition 1-2 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.638]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3
> > successfully
> > > > owned partition 1-3 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.646]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5
> > successfully
> > > > owned partition 1-5 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.656]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0
> > successfully
> > > > owned partition 1-0 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.660]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9
> > successfully
> > > > owned partition 1-9 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.665]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6
> > successfully
> > > > owned partition 1-6 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.669]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7
> > successfully
> > > > owned partition 1-7 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.674]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1
> > successfully
> > > > owned partition 1-1 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.679]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4
> > successfully
> > > > owned partition 1-4 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.684]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8
> > successfully
> > > > owned partition 1-8 for topic sensei]
> > > > [INFO  2014-05-13 16:19:09.684]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating
> > the
> > > > cache]
> > > > [INFO  2014-05-13 16:19:09.684]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
> > > > RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
> > > > partitions : sensei:1-0: fetched offset = 1555136587: consumed
> offset =
> > > > 1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
> > > > 1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
> > > > 1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
> > > > 1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
> > > > 10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
> > > > 1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
> > > > 1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
> > > > 1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
> > > > 1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
> > > > 1558487487]
> > > > [INFO  2014-05-13 16:19:09.684]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end
> > > rebalancing
> > > > consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
> try
> > > #0]
> > > > [INFO  2014-05-13 16:19:09.685]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 6 offset:
> > 1563630746
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.685]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 7 offset:
> > 1552261483
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.685]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 4 offset:
> 10316617
> > > from
> > > > 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.685]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 5 offset:
> > 1550214323
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.686]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 2 offset:
> > 1551158177
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.686]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 3 offset:
> > 1552939302
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.686]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 1 offset:
> > 1550960225
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.686]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 8 offset:
> > 1556683052
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.686]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 0 offset:
> > 1555136587
> > > > from 10.101.10.12:9092]
> > > > [INFO  2014-05-13 16:19:09.686]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [FetchRunnable-0 start fetching topic: sensei part: 9 offset:
> > 1558487487
> > > > from 10.101.10.12:9092]
> > > > [ERROR 2014-05-13 16:19:18.687]
> > > > kafka.utils.Logging$class.error(Logging.scala:89) [error in
> > > FetcherRunnable
> > > > ]
> > > >     at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:49)
> > > >     at
> > > >
> > >
> >
> kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:186)
> > > >     at
> > kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:113)
> > > >     at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
> > > > [INFO  2014-05-13 16:19:18.687]
> > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > [stopping fetcher FetchRunnable-0 to host 10.101.10.12]
> > > >
> > > >
> > > >
> > > > 2014-05-12 23:22 GMT+08:00 Guozhang Wang <wangguoz@gmail.com>:
> > > >
> > > > > Hello Yonghui,
> > > > >
> > > > > In 0.7 the consumer rebalance logic is distributed and in some
> corner
> > > > cases
> > > > > such as soft-failure-caused-consecutive rebalances some consumer
> may
> > > > > consider the rebalance as complete while others are still trying
> the
> > > > > rebalance process. You can check the GC logs on your consumer to
> > verify
> > > > if
> > > > > that is the case:
> > > > >
> > > > > https://issues.apache.org/jira/browse/KAFKA-242
> > > > >
> > > > > If you bounce the consumers to trigger another rebalance, this
> issue
> > > > would
> > > > > likely to be resolved.
> > > > >
> > > > > To solve this issue in 0.9 we are moving the group management like
> > load
> > > > > rebalance from the ZK-based distributed logic into a centralized
> > > > > coordiantor. Details can be found here:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Mon, May 12, 2014 at 12:48 AM, Yonghui Zhao <
> > zhaoyonghui@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We are using kafka 0.7.
> > > > > >
> > > > > > 2 brokers, each broker has 10 partitions for one topic
> > > > > > 3 consumers in one consumer group, each consumer create 10
> streams.
> > > > > >
> > > > > >
> > > > > > Today, when we want to rollout new service.
> > > > > > After we restart one consumer we find exceptions and warning.
> > > > > >
> > > > > > kafka.common.ConsumerRebalanceFailedException:
> > > > > > RecommendEvent_sd-sns-relation01.bj-1399630465426-53d3aefc can't
> > > > > rebalance
> > > > > > after 4 retries
> > > > > >
> > > > > >
> > > > > > [INFO  2014-05-12 15:17:47.364]
> > > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > > [conflict in /consumers/RecommendEvent/owners/sensei/1-2 data:
> > > > > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-2
> stored
> > > > data:
> > > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > > > > > [INFO  2014-05-12 15:17:47.366]
> > > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e
> waiting
> > > for
> > > > > the
> > > > > > partition ownership to be deleted: 1-2]
> > > > > > [INFO  2014-05-12 15:17:47.375]
> > > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > > [conflict in /consumers/RecommendEvent/owners/sensei/1-3 data:
> > > > > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-3
> stored
> > > > data:
> > > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > > > > > [INFO  2014-05-12 15:17:47.375]
> > > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e
> waiting
> > > for
> > > > > the
> > > > > > partition ownership to be deleted: 1-3]
> > > > > > [INFO  2014-05-12 15:17:47.385]
> > > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > > [conflict in /consumers/RecommendEvent/owners/sensei/1-5 data:
> > > > > > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-5
> stored
> > > > data:
> > > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2]
> > > > > > [INFO  2014-05-12 15:17:47.386]
> > > > > > kafka.utils.Logging$class.info(Logging.scala:61)
> > > > > > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e
> waiting
> > > for
> > > > > the
> > > > > > partition ownership to be deleted: 1-5]
> > > > > >
> > > > > >
> > > > > >
> > > > > > And I opened zk viewer.
> > > > > >
> > > > > > In zk, we found 2 consumers in ConsumerGroup/ids:
> > > > > >
> > > > > > RecommendEvent_sd-sns-relation02.bj-1399635256619-5d8123c6
> > > > > > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3
> > > > > >
> > > > > >
> > > > > > And in owners/topic/ we found all partitions are assigned to
> > > > > > sd-sns-relation03.bj:
> > > > > >
> > > > > > Here is the owner info:
> > > > > > 1:0  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > > > > 1:1  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > > > > 1:2  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > > > > 1:3  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > > > > 1:4  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > > > > 1:5  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > > > > 1:6  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > > > > 1:7  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > > > > 1:8  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > > > > 1:9  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > > > >
> > > > > > 2:0  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > > > > > 2:1  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > > > > > 2:2  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > > > > > 2:3  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > > > > > 2:4  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > > > > > 2:5  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-5
> > > > > > 2:6  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-6
> > > > > > 2:7  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-7
> > > > > > 2:8  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-8
> > > > > > 2:9  RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-9
> > > > > >
> > > > > >
> > > > > > So all partitions are assigned to sd-sns-relation03.bj,  but
> from
> > > logs
> > > > > and
> > > > > > counter, we are sure sd-sns-relation02.bj has input too.
> > > > > >
> > > > > >
> > > > > > My question is:
> > > > > >
> > > > > > 1. why rebalance failed?
> > > > > > 2. why owner info is wrong?  btw: zkclient is 0.2
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> >
> > --
> >
> >
> > *Best Regards 向河林*
> >
>
> --
>
>
> *Best Regards 向河林*
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message