kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Drew Goya <d...@gradientx.com>
Subject Re: Consumer Group Rebalance Issues
Date Mon, 23 Dec 2013 19:04:02 GMT
Thanks, I migrated our ZK cluster over to 3.3 this weekend.  Hopefully that
does it!


On Fri, Dec 20, 2013 at 9:09 AM, Jun Rao <junrao@gmail.com> wrote:

> Hmm, not sure how stable 3.4.4. We have been using 3.3.4 and haven't seen
> issues with ZK as long as there aren't many ZK session expirations.
>
> Thanks,
>
> Jun
>
>
> On Thu, Dec 19, 2013 at 9:41 PM, Drew Goya <drew@gradientx.com> wrote:
>
> > Our cluster is currently running 3.4.4.
> >
> > I see Kafka is currently using the 3.3.4 client, is there a potential
> > conflict there?
> >
> >
> > On Wed, Dec 18, 2013 at 9:12 PM, Jun Rao <junrao@gmail.com> wrote:
> >
> > > The issue is that consumer 007 didn't see consumer 006 during
> > rebalancing.
> > > So, it made a decision in conflict with consumer 006. Consumer 007
> should
> > > have another ZK watcher fired to trigger another rebalance when if it
> > will
> > > see consumer 006. Which version of ZK are you using?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, Dec 18, 2013 at 9:38 AM, Drew Goya <drew@gradientx.com> wrote:
> > >
> > > > Thanks for the help with this Jun, really appreciate it!  So I found
> > this
> > > > in the logs for consumer 007 about an hour previous.  Besides that no
> > > real
> > > > activity.
> > > >
> > > > It looks like 007 rebalanced and successfully claimed partition
> 24-27.
> > > >  Shortly after that its zookeeper client timed out and reconnected.
>  It
> > > > didn't rebalance again after this.
> > > >
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], begin
> > > > rebalancing consumer
> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8 try #0
> > > > 2013-12-17 15:51:06 ConsumerFetcherManager [INFO]
> > > > [ConsumerFetcherManager-1387249529483] Stopping leader finder thread
> > > > 2013-12-17 15:51:06 ConsumerFetcherManager$LeaderFinderThread [INFO]
> > > >
> > > >
> > >
> >
> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread],
> > > > Shutting down
> > > > 2013-12-17 15:51:06 ConsumerFetcherManager$LeaderFinderThread [INFO]
> > > >
> > > >
> > >
> >
> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread],
> > > > Stopped
> > > > 2013-12-17 15:51:06 ConsumerFetcherManager$LeaderFinderThread [INFO]
> > > >
> > > >
> > >
> >
> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread],
> > > > Shutdown completed
> > > > 2013-12-17 15:51:06 ConsumerFetcherManager [INFO]
> > > > [ConsumerFetcherManager-1387249529483] Stopping all fetchers
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-13],
> > > > Shutting down
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-13],
> > > > Stopped
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-13],
> > > > Shutdown completed
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-11],
> > > > Shutting down
> > > > 2013-12-17 15:51:06 SimpleConsumer [INFO] Reconnect due to socket
> > error:
> > > > null
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-11],
> > > > Stopped
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-11],
> > > > Shutdown completed
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-10],
> > > > Shutting down
> > > > 2013-12-17 15:51:06 SimpleConsumer [INFO] Reconnect due to socket
> > error:
> > > > null
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-10],
> > > > Stopped
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-10],
> > > > Shutdown completed
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-12],
> > > > Shutting down
> > > > 2013-12-17 15:51:06 SimpleConsumer [INFO] Reconnect due to socket
> > error:
> > > > null
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-12],
> > > > Stopped
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-12],
> > > > Shutdown completed
> > > > 2013-12-17 15:51:06 ConsumerFetcherManager [INFO]
> > > > [ConsumerFetcherManager-1387249529483] All connections stopped
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Cleared
> > all
> > > > relevant queues for this fetcher
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Cleared
> > the
> > > > data chunks in all the consumer message iterators
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8],
> > Committing
> > > > all offsets after clearing the fetcher queues
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8],
> Releasing
> > > > partition ownership
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8],
> Consumer
> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8
> rebalancing
> > > the
> > > > following partitions: List(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
> > 13,
> > > > 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
> 31,
> > > 32,
> > > > 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49,
> 50,
> > > 51,
> > > > 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68,
> 69,
> > > 70,
> > > > 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87,
> 88,
> > > 89,
> > > > 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105,
> > > 106,
> > > > 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120,
> > > 121,
> > > > 122, 123, 124, 125, 126, 127) for topic Events2 with consumers:
> > > > List(trackingGroup_prod-storm-sup-trk001-1387249529775-2a8484f1-0,
> > > > trackingGroup_prod-storm-sup-trk001-1387249529775-2a8484f1-1,
> > > > trackingGroup_prod-storm-sup-trk002-1387249530831-97c586ab-0,
> > > > trackingGroup_prod-storm-sup-trk002-1387249530831-97c586ab-1,
> > > > trackingGroup_prod-storm-sup-trk003-1387249529739-f2de3dd9-0,
> > > > trackingGroup_prod-storm-sup-trk003-1387249529739-f2de3dd9-1,
> > > > trackingGroup_prod-storm-sup-trk004-1387249530445-8f57ec5c-0,
> > > > trackingGroup_prod-storm-sup-trk004-1387249530445-8f57ec5c-1,
> > > > trackingGroup_prod-storm-sup-trk005-1387249530451-d59c669a-0,
> > > > trackingGroup_prod-storm-sup-trk005-1387249530451-d59c669a-1,
> > > > trackingGroup_prod-storm-sup-trk005-1387249530452-2b244683-0,
> > > > trackingGroup_prod-storm-sup-trk005-1387249530452-2b244683-1,
> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0,
> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1,
> > > > trackingGroup_prod-storm-sup-trk008-1387249526700-11ba655b-0,
> > > > trackingGroup_prod-storm-sup-trk008-1387249526700-11ba655b-1,
> > > > trackingGroup_prod-storm-sup-trk009-1387249530020-cb36831c-0,
> > > > trackingGroup_prod-storm-sup-trk009-1387249530020-cb36831c-1,
> > > > trackingGroup_prod-storm-sup-trk010-1387249529975-d43aff06-0,
> > > > trackingGroup_prod-storm-sup-trk010-1387249529975-d43aff06-1,
> > > > trackingGroup_prod-storm-sup-trk011-1387249527684-479a04f9-0,
> > > > trackingGroup_prod-storm-sup-trk011-1387249527684-479a04f9-1,
> > > > trackingGroup_prod-storm-sup-trk012-1387249530208-155ecd68-0,
> > > > trackingGroup_prod-storm-sup-trk012-1387249530208-155ecd68-1,
> > > > trackingGroup_prod-storm-sup-trk013-1387249530700-b323ee53-0,
> > > > trackingGroup_prod-storm-sup-trk013-1387249530700-b323ee53-1,
> > > > trackingGroup_prod-storm-sup-trk014-1387249529916-e32e6363-0,
> > > > trackingGroup_prod-storm-sup-trk014-1387249529916-e32e6363-1,
> > > > trackingGroup_prod-storm-sup-trk015-1387249529709-d655ccd4-0,
> > > > trackingGroup_prod-storm-sup-trk015-1387249529709-d655ccd4-1,
> > > > trackingGroup_prod-storm-sup-trk016-1387249531064-bc8f8f3e-0,
> > > > trackingGroup_prod-storm-sup-trk016-1387249531064-bc8f8f3e-1,
> > > > trackingGroup_prod-storm-sup-trk017-1387249530635-35f505b7-0,
> > > > trackingGroup_prod-storm-sup-trk017-1387249530635-35f505b7-1,
> > > > trackingGroup_prod-storm-sup-trk018-1387249530621-84327f5f-0,
> > > > trackingGroup_prod-storm-sup-trk018-1387249530621-84327f5f-1,
> > > > trackingGroup_prod-storm-sup-trk019-1387249530418-80afccf9-0,
> > > > trackingGroup_prod-storm-sup-trk019-1387249530418-80afccf9-1,
> > > > trackingGroup_prod-storm-sup-trk020-1387249530930-906e99e1-0,
> > > > trackingGroup_prod-storm-sup-trk020-1387249530930-906e99e1-1,
> > > > trackingGroup_prod-storm-sup-trk021-1387249529761-705a5bca-0,
> > > > trackingGroup_prod-storm-sup-trk021-1387249529761-705a5bca-1,
> > > > trackingGroup_prod-storm-sup-trk022-1387249530347-3d40b4f9-0,
> > > > trackingGroup_prod-storm-sup-trk022-1387249530347-3d40b4f9-1,
> > > > trackingGroup_prod-storm-sup-trk023-1387249529067-957d280b-0,
> > > > trackingGroup_prod-storm-sup-trk023-1387249529067-957d280b-1,
> > > > trackingGroup_prod-storm-sup-trk024-1387249530625-f8118f02-0,
> > > > trackingGroup_prod-storm-sup-trk024-1387249530625-f8118f02-1,
> > > > trackingGroup_prod-storm-sup-trk025-1387249530213-cccfffc8-0,
> > > > trackingGroup_prod-storm-sup-trk025-1387249530213-cccfffc8-1,
> > > > trackingGroup_prod-storm-sup-trk046-1387249527798-2164c569-0,
> > > > trackingGroup_prod-storm-sup-trk046-1387249527798-2164c569-1,
> > > > trackingGroup_prod-storm-sup-trk047-1387249530559-6b49ce74-0,
> > > > trackingGroup_prod-storm-sup-trk047-1387249530559-6b49ce74-1,
> > > > trackingGroup_prod-storm-sup-trk048-1387249529976-aba1e428-0,
> > > > trackingGroup_prod-storm-sup-trk048-1387249529976-aba1e428-1,
> > > > trackingGroup_prod-storm-sup-trk050-1387249530465-dc203a62-0,
> > > > trackingGroup_prod-storm-sup-trk050-1387249530465-dc203a62-1,
> > > > trackingGroup_prod-storm-sup-trk051-1387249530406-46f7a649-0,
> > > > trackingGroup_prod-storm-sup-trk051-1387249530406-46f7a649-1,
> > > > trackingGroup_prod-storm-sup-trk052-1387249530423-e06e4210-0,
> > > > trackingGroup_prod-storm-sup-trk052-1387249530423-e06e4210-1,
> > > > trackingGroup_prod-storm-sup-trk054-1387249530369-68e494e6-0,
> > > > trackingGroup_prod-storm-sup-trk054-1387249530369-68e494e6-1,
> > > > trackingGroup_prod-storm-sup-trk055-1387249529961-bec0abbc-0,
> > > > trackingGroup_prod-storm-sup-trk055-1387249529961-bec0abbc-1,
> > > > trackingGroup_prod-storm-sup-trk056-1387249531590-957c0b49-0,
> > > > trackingGroup_prod-storm-sup-trk056-1387249531590-957c0b49-1,
> > > > trackingGroup_prod-storm-sup-trk057-1387249530341-d8476874-0,
> > > > trackingGroup_prod-storm-sup-trk057-1387249530341-d8476874-1,
> > > > trackingGroup_prod-storm-sup-trk058-1387249530730-20554b4d-0,
> > > > trackingGroup_prod-storm-sup-trk058-1387249530730-20554b4d-1)
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8],
> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0
> attempting
> > > to
> > > > claim partition 24
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8],
> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0
> attempting
> > > to
> > > > claim partition 25
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8],
> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1
> attempting
> > > to
> > > > claim partition 26
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8],
> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1
> attempting
> > > to
> > > > claim partition 27
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8],
> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0
> > successfully
> > > > owned partition 25 for topic Events2
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8],
> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1
> > successfully
> > > > owned partition 26 for topic Events2
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8],
> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0
> > successfully
> > > > owned partition 24 for topic Events2
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8],
> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1
> > successfully
> > > > owned partition 27 for topic Events2
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8],
> Updating
> > > the
> > > > cache
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8],
> Consumer
> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8 selected
> > > > partitions : Events2:24: fetched offset = 969809475: consumed offset
> =
> > > > 969809475,Events2:25: fetched offset = 983792923: consumed offset =
> > > > 983792923,Events2:26: fetched offset = 90409778: consumed offset =
> > > > 90409778,Events2:27: fetched offset = 979456347: consumed offset =
> > > > 979456347
> > > > 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
> > > > [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], end
> > > > rebalancing consumer
> > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8 try #0
> > > > 2013-12-17 15:51:06 ConsumerFetcherManager$LeaderFinderThread [INFO]
> > > >
> > > >
> > >
> >
> [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread],
> > > > Starting
> > > > 2013-12-17 15:51:06 VerifiableProperties [INFO] Verifying properties
> > > > 2013-12-17 15:51:06 VerifiableProperties [INFO] Property client.idis
> > > > overridden to trackingGroup
> > > > 2013-12-17 15:51:06 VerifiableProperties [INFO] Property
> > > > metadata.broker.list is overridden to
> > > >
> > > >
> > >
> >
> prod-kafka-broker001:9092,prod-kafka-broker010:9092,prod-kafka-broker011:9092,prod-kafka-broker012:9092,prod-kafka-broker013:9092,prod-kafka-broker014:9092,prod-kafka-broker015:9092,prod-kafka-broker002:9092,prod-kafka-broker003:9092,prod-kafka-broker004:9092,prod-kafka-broker005:9092,prod-kafka-broker006:9092,prod-kafka-broker007:9092,prod-kafka-broker008:9092,prod-kafka-broker009:9092
> > > > 2013-12-17 15:51:06 VerifiableProperties [INFO] Property
> > > > request.timeout.msis overridden to 30000
> > > > 2013-12-17 15:51:06 ClientUtils$ [INFO] Fetching metadata from broker
> > > > id:15,host:prod-kafka-broker015,port:9092 with correlation id 23 for
> 1
> > > > topic(s) Set(Events2)
> > > > 2013-12-17 15:51:06 SyncProducer [INFO] Connected to
> > > > prod-kafka-broker015:9092 for producing
> > > > 2013-12-17 15:51:06 SyncProducer [INFO] Disconnecting from
> > > > prod-kafka-broker015:9092
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-3],
> > > > Starting
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-4],
> > > > Starting
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-9],
> > > > Starting
> > > > 2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
> > > >
> > > >
> > >
> >
> [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-2],
> > > > Starting
> > > > 2013-12-17 15:51:06 ConsumerFetcherManager [INFO]
> > > > [ConsumerFetcherManager-1387249529483] Added fetcher for partitions
> > > > ArrayBuffer([[Events2,25], initOffset 983792923 to broker
> > > > id:3,host:prod-kafka-broker003,port:9092] , [[Events2,26], initOffset
> > > > 90409778 to broker id:4,host:prod-kafka-broker004,port:9092] ,
> > > > [[Events2,24], initOffset 969809475 to broker
> > > > id:2,host:prod-kafka-broker002,port:9092] , [[Events2,27], initOffset
> > > > 979456347 to broker id:9,host:prod-kafka-broker009,port:9092] )
> > > > 2013-12-17 15:51:15 executor [ERROR]
> > > > 2013-12-17 15:51:15 executor [ERROR]
> > > > 2013-12-17 15:51:15 ClientCnxn [INFO] Client session timed out, have
> > not
> > > > heard from server in 5015ms for sessionid 0x342e4febc180841, closing
> > > socket
> > > > connection and attempting reconnect
> > > > 2013-12-17 15:51:15 ZkClient [INFO] zookeeper state changed
> > > (Disconnected)
> > > > 2013-12-17 15:51:16 ClientCnxn [INFO] Opening socket connection to
> > server
> > > > prod-zookeeper-kafka002/10.4.34.186:2181
> > > > 2013-12-17 15:51:16 ClientCnxn [INFO] Socket connection established
> to
> > > > prod-zookeeper-kafka002/10.4.34.186:2181, initiating session
> > > > 2013-12-17 15:51:16 ClientCnxn [INFO] Session establishment complete
> on
> > > > server prod-zookeeper-kafka002/10.4.34.186:2181, sessionid =
> > > > 0x342e4febc180841, negotiated timeout = 6000
> > > > 2013-12-17 15:51:16 ZkClient [INFO] zookeeper state changed
> > > (SyncConnected)
> > > > 2013-12-17 15:51:16 executor [ERROR]
> > > > 2013-12-17 15:51:20 executor [ERROR]
> > > > 2013-12-17 15:51:24 executor [ERROR]
> > > >
> > > >
> > > > On Tue, Dec 17, 2013 at 9:24 PM, Jun Rao <junrao@gmail.com> wrote:
> > > >
> > > > > What's consumer trackingGroup_prod-storm-sup-trk007 doing at the
> same
> > > > time?
> > > > > It's the one that caused the conflict in ZK.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Tue, Dec 17, 2013 at 9:19 PM, Drew Goya <drew@gradientx.com>
> > wrote:
> > > > >
> > > > > > I explored that possibility but I'm not seeing any ZK session
> > > > expirations
> > > > > > in the logs and it doesn't look like these rebalances complete.
> > > > > >
> > > > > > They fail due to conflicts in the zookeeper data
> > > > > >
> > > > > >
> > > > > > On Tue, Dec 17, 2013 at 8:53 PM, Jun Rao <junrao@gmail.com>
> wrote:
> > > > > >
> > > > > > > Have you looked at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog
> > > > > > > ?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Dec 17, 2013 at 9:24 AM, Drew Goya <drew@gradientx.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Hey all,
> > > > > > > >
> > > > > > > > I've recently been having problems with consumer groups
> > > > rebalancing.
> > > > > >  I'm
> > > > > > > > using several high level consumers which all belong
to the
> same
> > > > > group.
> > > > > > > >  Occasionally one or two of them will get stuck in
a
> rebalance
> > > > loop.
> > > > > > >  They
> > > > > > > > attempt to rebalance, but the partitions they try
to claim
> are
> > > > owned.
> > > > > > > >  Anyone run into this?  Ideas?
> > > > > > > >
> > > > > > > > I see errors in my zookeeper logs like:
> > > > > > > >
> > > > > > > > 2013-12-17 17:12:31,171 [myid:001] - INFO
>  [ProcessThread(sid:1
> > > > > > > > cport:-1)::PrepRequestProcessor@627] - Got user-level
> > > > > KeeperException
> > > > > > > when
> > > > > > > > processing sessionid:0x342e4febc180852 type:create
> cxid:0x1a9a
> > > > > > > > zxid:0x501390d4b txntype:-1 reqpath:n/a Error
> > > > > > > > Path:/kafka/consumers/trackingGroup/owners/Events2/25
> > > > > > > Error:KeeperErrorCode
> > > > > > > > = NodeExists for
> > /kafka/consumers/trackingGroup/owners/Events2/25
> > > > > > > >
> > > > > > > > And errors in my kafka logs like:
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:32 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > begin
> > > > > > > > rebalancing consumer
> > > > > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306
> try
> > #8
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO]
> > > > > > > > [ConsumerFetcherManager-1387249530381] Stopping leader
finder
> > > > thread
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO]
> > > > > > > > [ConsumerFetcherManager-1387249530381] Stopping all
fetchers
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO]
> > > > > > > > [ConsumerFetcherManager-1387249530381] All connections
> stopped
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > Cleared
> > > > > > all
> > > > > > > > relevant queues for this fetcher
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > Cleared
> > > > > > the
> > > > > > > > data chunks in all the consumer message iterators
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > > > Committing
> > > > > > > > all offsets after clearing the fetcher queues
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > > Releasing
> > > > > > > > partition ownership
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > > Consumer
> > > > > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306
> > > > > rebalancing
> > > > > > > the
> > > > > > > > following partitions: List(0, 1, 2, 3, 4, 5, 6, 7,
8, 9, 10,
> > 11,
> > > > 12,
> > > > > > 13,
> > > > > > > > 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26,
27, 28,
> 29,
> > > 30,
> > > > > 31,
> > > > > > > 32,
> > > > > > > > 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45,
46, 47,
> 48,
> > > 49,
> > > > > 50,
> > > > > > > 51,
> > > > > > > > 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64,
65, 66,
> 67,
> > > 68,
> > > > > 69,
> > > > > > > 70,
> > > > > > > > 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83,
84, 85,
> 86,
> > > 87,
> > > > > 88,
> > > > > > > 89,
> > > > > > > > 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101,
102, 103,
> > 104,
> > > > 105,
> > > > > > > 106,
> > > > > > > > 107, 108, 109, 110, 111, 112, 113, 114, 115, 116,
117, 118,
> > 119,
> > > > 120,
> > > > > > > 121,
> > > > > > > > 122, 123, 124, 125, 126, 127) for topic Events2 with
> consumers:
> > > > > > > >
> > > List(trackingGroup_prod-storm-sup-trk001-1387249529775-2a8484f1-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk001-1387249529775-2a8484f1-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk002-1387249530831-97c586ab-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk002-1387249530831-97c586ab-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk003-1387249529739-f2de3dd9-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk003-1387249529739-f2de3dd9-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk004-1387249530445-8f57ec5c-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk004-1387249530445-8f57ec5c-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk005-1387249530451-d59c669a-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk005-1387249530451-d59c669a-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk005-1387249530452-2b244683-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk005-1387249530452-2b244683-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk008-1387249526700-11ba655b-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk008-1387249526700-11ba655b-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk009-1387249530020-cb36831c-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk009-1387249530020-cb36831c-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk010-1387249529975-d43aff06-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk010-1387249529975-d43aff06-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk011-1387249527684-479a04f9-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk011-1387249527684-479a04f9-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk012-1387249530208-155ecd68-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk012-1387249530208-155ecd68-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk013-1387249530700-b323ee53-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk013-1387249530700-b323ee53-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk014-1387249529916-e32e6363-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk014-1387249529916-e32e6363-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk015-1387249529709-d655ccd4-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk015-1387249529709-d655ccd4-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk016-1387249531064-bc8f8f3e-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk016-1387249531064-bc8f8f3e-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk017-1387249530635-35f505b7-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk017-1387249530635-35f505b7-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk018-1387249530621-84327f5f-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk018-1387249530621-84327f5f-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk019-1387249530418-80afccf9-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk019-1387249530418-80afccf9-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk020-1387249530930-906e99e1-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk020-1387249530930-906e99e1-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk021-1387249529761-705a5bca-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk021-1387249529761-705a5bca-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk022-1387249530347-3d40b4f9-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk022-1387249530347-3d40b4f9-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk023-1387249529067-957d280b-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk023-1387249529067-957d280b-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk024-1387249530625-f8118f02-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk024-1387249530625-f8118f02-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk025-1387249530213-cccfffc8-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk025-1387249530213-cccfffc8-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk046-1387249527798-2164c569-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk046-1387249527798-2164c569-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk047-1387249530559-6b49ce74-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk047-1387249530559-6b49ce74-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk048-1387249529976-aba1e428-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk048-1387249529976-aba1e428-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk050-1387249530465-dc203a62-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk050-1387249530465-dc203a62-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk051-1387249530406-46f7a649-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk051-1387249530406-46f7a649-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk052-1387249530423-e06e4210-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk052-1387249530423-e06e4210-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk054-1387249530369-68e494e6-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk054-1387249530369-68e494e6-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk055-1387249529961-bec0abbc-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk055-1387249529961-bec0abbc-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk056-1387249531590-957c0b49-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk056-1387249531590-957c0b49-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk057-1387249530341-d8476874-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk057-1387249530341-d8476874-1,
> > > > > > > > trackingGroup_prod-storm-sup-trk058-1387249530730-20554b4d-0,
> > > > > > > > trackingGroup_prod-storm-sup-trk058-1387249530730-20554b4d-1)
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-0
> > > > > attempting
> > > > > > > to
> > > > > > > > claim partition 24
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-0
> > > > > attempting
> > > > > > > to
> > > > > > > > claim partition 25
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-1
> > > > > attempting
> > > > > > > to
> > > > > > > > claim partition 26
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-1
> > > > > attempting
> > > > > > > to
> > > > > > > > claim partition 27
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZkUtils$ [INFO] conflict in
> > > > > > > > /consumers/trackingGroup/owners/Events2/25 data:
> > > > > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-0
> > > stored
> > > > > > data:
> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > waiting
> > > > > > for
> > > > > > > > the partition ownership to be deleted: 25
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZkUtils$ [INFO] conflict in
> > > > > > > > /consumers/trackingGroup/owners/Events2/26 data:
> > > > > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-1
> > > stored
> > > > > > data:
> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > waiting
> > > > > > for
> > > > > > > > the partition ownership to be deleted: 26
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZkUtils$ [INFO] conflict in
> > > > > > > > /consumers/trackingGroup/owners/Events2/24 data:
> > > > > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-0
> > > stored
> > > > > > data:
> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > waiting
> > > > > > for
> > > > > > > > the partition ownership to be deleted: 24
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZkUtils$ [INFO] conflict in
> > > > > > > > /consumers/trackingGroup/owners/Events2/27 data:
> > > > > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306-1
> > > stored
> > > > > > data:
> > > > > > > > trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-1
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > waiting
> > > > > > for
> > > > > > > > the partition ownership to be deleted: 27
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > end
> > > > > > > > rebalancing consumer
> > > > > > > > trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306
> try
> > #8
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > > > Rebalancing
> > > > > > > > attempt failed. Clearing the cache before the next
> rebalancing
> > > > > > operation
> > > > > > > is
> > > > > > > > triggered
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO]
> > > > > > > > [ConsumerFetcherManager-1387249530381] Stopping leader
finder
> > > > thread
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO]
> > > > > > > > [ConsumerFetcherManager-1387249530381] Stopping all
fetchers
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ConsumerFetcherManager [INFO]
> > > > > > > > [ConsumerFetcherManager-1387249530381] All connections
> stopped
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > Cleared
> > > > > > all
> > > > > > > > relevant queues for this fetcher
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > Cleared
> > > > > > the
> > > > > > > > data chunks in all the consumer message iterators
> > > > > > > >
> > > > > > > > 2013-12-17 17:20:33 ZookeeperConsumerConnector [INFO]
> > > > > > > > [trackingGroup_prod-storm-sup-trk006-1387249530327-9d15c306],
> > > > > > Committing
> > > > > > > > all offsets after clearing the fetcher queues
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message