kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Navneeth Krishnan <reachnavnee...@gmail.com>
Subject Kafka Streams - Source Partition Assignment Issue
Date Mon, 14 Dec 2020 19:29:01 GMT
Hi All,

How does kafka streams partition assignment work for sources? I have a
stream application reading from a topic which has 24 partitions. There are
6 application containers with 4 stream tasks in each container running but
only 2 instances are assigned with partitions and even within the two, one
has 21 partitions and the other has 3 partitions assigned. There is
something wrong in the way the partitions are assigned. Is there a way to
evenly distribute partitions?

*Group Assignment:*
userapp     user-data               5          1716495057      1716504638
   9581
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
userapp     user-data               13         1735721509      1735730754
   9245
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
userapp     user-data               9          1729975595      1729984597
   9002
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
userapp     user-data               17         1736457935      1736466997
   9062
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
userapp     user-data               0          1917953948      1917963103
   9155
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
userapp     user-data               11         1719082482      1719085979
   3497
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
userapp     user-data               19         1755067810      1755071386
   3576
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
userapp     user-data               7          1729598367      1729601982
   3615
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
userapp     user-data               1          1949943357      1949947252
   3895
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
userapp     user-data               15         1751889619      1751893116
   3497
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
userapp     user-data               3          1752352594      1752356255
   3661
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
userapp     user-data               6          1693612741      1693621514
   8773
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
userapp     user-data               2          1923482321      1923490826
   8505
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
userapp     user-data               14         1677960791      1677969039
   8248
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
userapp     user-data               10         1722133802      1722142388
   8586
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
userapp     user-data               18         1739152927      1739161360
   8433
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
userapp     user-data               20         1715428183      1715431991
   3808
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
userapp     user-data               8          1738854860      1738858541
   3681
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
userapp     user-data               4          1704061352      1704065142
   3790
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
userapp     user-data               12         1743370130      1743373727
   3597
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
userapp     user-data               16         1696027581      1696031399
   3818
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
/172.21.82.140
 userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
userapp     user-data               21         1722643635      1722647766
   4131
 userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer-19fdbf9f-fb65-45fb-a020-aa16e9da76c5
/172.21.64.86
userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer
userapp     user-data               23         1675792619      1675800530
   7911
 userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer-e98dae60-89b9-44b0-a366-d533b288fcf8
/172.21.64.86
userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer
userapp     user-data               22         1735482901      1735490720
   7819
 userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer-3553c1a0-def4-453d-a4fb-a6c4f08ad3f2
/172.21.64.86
userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer

Regards,
Navneeth

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message