kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Navneeth Krishnan <reachnavnee...@gmail.com>
Subject Re: Kafka Streams - Source Partition Assignment Issue
Date Tue, 15 Dec 2020 00:07:01 GMT
Thanks Guozhang for the suggestion.

We are using kafka 2.3.0 and the app.id is set to the same value.

Bouncing off instances work for a small period of time but when auto
scaling happens it skews it up again. This is becoming a big bottleneck for
us since only one node is now trying to handle more than 60% of traffic and
we are getting into huge lag.

Regards,
Navneeth

On Mon, Dec 14, 2020 at 2:49 PM Guozhang Wang <wangguoz@gmail.com> wrote:

> Kafka Streams should evenly distribute the partitions, but there are some
> issues in old versions of Kafka that you may be observing. To verify if it
> is a transient issue or it is permanently, I'd suggest you try:
>
> 1) bounce the instances that have no partitions assigned (not bounce them
> all together, but in rolling manner), and see if after the rebalance they
> are evenly distributed.
>
> 2) check if the app.id are set to the same value.
>
> Also which version of Kafka are you using?
>
>
> Guozhang
>
>
>
>
> On Mon, Dec 14, 2020 at 11:29 AM Navneeth Krishnan <
> reachnavneeth2@gmail.com>
> wrote:
>
> > Hi All,
> >
> > How does kafka streams partition assignment work for sources? I have a
> > stream application reading from a topic which has 24 partitions. There
> are
> > 6 application containers with 4 stream tasks in each container running
> but
> > only 2 instances are assigned with partitions and even within the two,
> one
> > has 21 partitions and the other has 3 partitions assigned. There is
> > something wrong in the way the partitions are assigned. Is there a way to
> > evenly distribute partitions?
> >
> > *Group Assignment:*
> > userapp     user-data               5          1716495057      1716504638
> >    9581
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> > userapp     user-data               13         1735721509      1735730754
> >    9245
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> > userapp     user-data               9          1729975595      1729984597
> >    9002
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> > userapp     user-data               17         1736457935      1736466997
> >    9062
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> > userapp     user-data               0          1917953948      1917963103
> >    9155
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
> > userapp     user-data               11         1719082482      1719085979
> >    3497
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp     user-data               19         1755067810      1755071386
> >    3576
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp     user-data               7          1729598367      1729601982
> >    3615
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp     user-data               1          1949943357      1949947252
> >    3895
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp     user-data               15         1751889619      1751893116
> >    3497
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp     user-data               3          1752352594      1752356255
> >    3661
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
> > userapp     user-data               6          1693612741      1693621514
> >    8773
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> > userapp     user-data               2          1923482321      1923490826
> >    8505
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> > userapp     user-data               14         1677960791      1677969039
> >    8248
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> > userapp     user-data               10         1722133802      1722142388
> >    8586
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> > userapp     user-data               18         1739152927      1739161360
> >    8433
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
> > userapp     user-data               20         1715428183      1715431991
> >    3808
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> > userapp     user-data               8          1738854860      1738858541
> >    3681
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> > userapp     user-data               4          1704061352      1704065142
> >    3790
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> > userapp     user-data               12         1743370130      1743373727
> >    3597
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> > userapp     user-data               16         1696027581      1696031399
> >    3818
> >
> >
> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
> > /172.21.82.140
> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
> > userapp     user-data               21         1722643635      1722647766
> >    4131
> >
> >
> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer-19fdbf9f-fb65-45fb-a020-aa16e9da76c5
> > /172.21.64.86
> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer
> > userapp     user-data               23         1675792619      1675800530
> >    7911
> >
> >
> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer-e98dae60-89b9-44b0-a366-d533b288fcf8
> > /172.21.64.86
> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer
> > userapp     user-data               22         1735482901      1735490720
> >    7819
> >
> >
> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer-3553c1a0-def4-453d-a4fb-a6c4f08ad3f2
> > /172.21.64.86
> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer
> >
> > Regards,
> > Navneeth
> >
>
>
> --
> -- Guozhang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message