Thanks Guozhang for the suggestion.
We are using kafka 2.3.0 and the app.id is set to the same value.
Bouncing off instances work for a small period of time but when auto
scaling happens it skews it up again. This is becoming a big bottleneck for
us since only one node is now trying to handle more than 60% of traffic and
we are getting into huge lag.
Regards,
Navneeth
On Mon, Dec 14, 2020 at 2:49 PM Guozhang Wang <wangguoz@gmail.com> wrote:
> Kafka Streams should evenly distribute the partitions, but there are some
> issues in old versions of Kafka that you may be observing. To verify if it
> is a transient issue or it is permanently, I'd suggest you try:
>
> 1) bounce the instances that have no partitions assigned (not bounce them
> all together, but in rolling manner), and see if after the rebalance they
> are evenly distributed.
>
> 2) check if the app.id are set to the same value.
>
> Also which version of Kafka are you using?
>
>
> Guozhang
>
>
>
>
> On Mon, Dec 14, 2020 at 11:29 AM Navneeth Krishnan <
> reachnavneeth2@gmail.com>
> wrote:
>
> > Hi All,
> >
> > How does kafka streams partition assignment work for sources? I have a
> > stream application reading from a topic which has 24 partitions. There
> are
> > 6 application containers with 4 stream tasks in each container running
> but
> > only 2 instances are assigned with partitions and even within the two,
> one
> > has 21 partitions and the other has 3 partitions assigned. There is
> > something wrong in the way the partitions are assigned. Is there a way to
> > evenly distribute partitions?
> >
> > *Group Assignment:*
> > userapp user-data 5 1716495057 1716504638
> > 9581
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> > userapp user-data 13 1735721509 1735730754
> > 9245
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> > userapp user-data 9 1729975595 1729984597
> > 9002
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> > userapp user-data 17 1736457935 1736466997
> > 9062
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> > userapp user-data 0 1917953948 1917963103
> > 9155
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> > userapp user-data 11 1719082482 1719085979
> > 3497
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp user-data 19 1755067810 1755071386
> > 3576
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp user-data 7 1729598367 1729601982
> > 3615
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp user-data 1 1949943357 1949947252
> > 3895
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp user-data 15 1751889619 1751893116
> > 3497
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp user-data 3 1752352594 1752356255
> > 3661
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp user-data 6 1693612741 1693621514
> > 8773
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> > userapp user-data 2 1923482321 1923490826
> > 8505
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> > userapp user-data 14 1677960791 1677969039
> > 8248
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> > userapp user-data 10 1722133802 1722142388
> > 8586
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> > userapp user-data 18 1739152927 1739161360
> > 8433
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> > userapp user-data 20 1715428183 1715431991
> > 3808
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> > userapp user-data 8 1738854860 1738858541
> > 3681
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> > userapp user-data 4 1704061352 1704065142
> > 3790
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> > userapp user-data 12 1743370130 1743373727
> > 3597
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> > userapp user-data 16 1696027581 1696031399
> > 3818
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> > /172.21.82.140
> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> > userapp user-data 21 1722643635 1722647766
> > 4131
> >
> >
> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer-19fdbf9f-fb65-45fb-a020-aa16e9da76c5
> > /172.21.64.86
> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer
> > userapp user-data 23 1675792619 1675800530
> > 7911
> >
> >
> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer-e98dae60-89b9-44b0-a366-d533b288fcf8
> > /172.21.64.86
> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer
> > userapp user-data 22 1735482901 1735490720
> > 7819
> >
> >
> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer-3553c1a0-def4-453d-a4fb-a6c4f08ad3f2
> > /172.21.64.86
> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer
> >
> > Regards,
> > Navneeth
> >
>
>
> --
> -- Guozhang
>
|