kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Navneeth Krishnan <reachnavnee...@gmail.com>
Subject Re: Kafka Streams - Source Partition Assignment Issue
Date Tue, 15 Dec 2020 08:03:49 GMT
In addition to the assignment issue I see the state of the application
changes every few minutes from Running to Rebalancing.

Other than the default configs only the below two extra config parameters
are set. Currently there are 10 instances and they are constantly
rebalancing.

fetch-max-wait: 3000
session.timeout.ms: 60000


2020-12-14T23:34:54.715-08:00
[userapp-ee690c43-9457-4c0d-a931-6a3819341e48-StreamThread-4] INFO State
transition from RUNNING to REBALANCING

2020-12-14T23:34:54.715-08:00
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b-StreamThread-3] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b] State transition from
RUNNING to REBALANCING

2020-12-14T23:34:54.715-08:00
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b-StreamThread-3] INFO State
transition from RUNNING to REBALANCING

2020-12-14T23:34:54.721-08:00
[userapp-77c70232-bd1e-4d7f-9543-2d5b6ccf2781-StreamThread-4] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-77c70232-bd1e-4d7f-9543-2d5b6ccf2781] State transition from
RUNNING to REBALANCING

2020-12-14T23:34:54.721-08:00
[userapp-77c70232-bd1e-4d7f-9543-2d5b6ccf2781-StreamThread-4] INFO State
transition from RUNNING to REBALANCING

2020-12-14T23:34:54.733-08:00
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a-StreamThread-4] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a] State transition from
RUNNING to REBALANCING

*2020-12-14T23:34:54.733-08:00*
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a-StreamThread-4] INFO State
transition from RUNNING to REBALANCING

*2020-12-14T23:34:57.898-08:00
*[userapp-edc066cc-2a51-4c92-988c-0b03f1015398-StreamThread-1]
INFO org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-edc066cc-2a51-4c92-988c-0b03f1015398] State transition from
RUNNING to REBALANCING

2020-12-14T23:34:57.898-08:00
[userapp-edc066cc-2a51-4c92-988c-0b03f1015398-StreamThread-1] INFO State
transition from RUNNING to REBALANCING

2020-12-14T23:34:57.899-08:00
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b-StreamThread-4] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b] State transition from
RUNNING to REBALANCING

2020-12-14T23:34:57.899-08:00
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b-StreamThread-4] INFO State
transition from RUNNING to REBALANCING

2020-12-14T23:34:57.899-08:00
[userapp-0f2af06e-e8f2-4dc2-b852-0a6484c7e4e9-StreamThread-3] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-0f2af06e-e8f2-4dc2-b852-0a6484c7e4e9] State transition from
RUNNING to REBALANCING

2020-12-14T23:34:57.899-08:00
[userapp-0f2af06e-e8f2-4dc2-b852-0a6484c7e4e9-StreamThread-3] INFO State
transition from RUNNING to REBALANCING

2020-12-14T23:34:57.900-08:00
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a-StreamThread-4] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a] State transition from
RUNNING to REBALANCING

2020-12-14T23:34:57.900-08:00
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a-StreamThread-4] INFO State
transition from RUNNING to REBALANCING

2020-12-14T23:34:57.900-08:00
[userapp-ee690c43-9457-4c0d-a931-6a3819341e48-StreamThread-3] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-ee690c43-9457-4c0d-a931-6a3819341e48] State transition from
RUNNING to REBALANCING

2020-12-14T23:34:57.900-08:00
[userapp-ee690c43-9457-4c0d-a931-6a3819341e48-StreamThread-3] INFO State
transition from RUNNING to REBALANCING

2020-12-14T23:34:57.900-08:00
[userapp-bd83fccf-8cc7-4f80-bd94-42cffd65faf5-StreamThread-2] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-bd83fccf-8cc7-4f80-bd94-42cffd65faf5] State transition from
RUNNING to REBALANCING

2020-12-14T23:34:57.900-08:00
[userapp-bd83fccf-8cc7-4f80-bd94-42cffd65faf5-StreamThread-2] INFO State
transition from RUNNING to REBALANCING

Thanks

On Mon, Dec 14, 2020 at 4:07 PM Navneeth Krishnan <reachnavneeth2@gmail.com>
wrote:

> Thanks Guozhang for the suggestion.
>
> We are using kafka 2.3.0 and the app.id is set to the same value.
>
> Bouncing off instances work for a small period of time but when auto
> scaling happens it skews it up again. This is becoming a big bottleneck for
> us since only one node is now trying to handle more than 60% of traffic and
> we are getting into huge lag.
>
> Regards,
> Navneeth
>
> On Mon, Dec 14, 2020 at 2:49 PM Guozhang Wang <wangguoz@gmail.com> wrote:
>
>> Kafka Streams should evenly distribute the partitions, but there are some
>> issues in old versions of Kafka that you may be observing. To verify if it
>> is a transient issue or it is permanently, I'd suggest you try:
>>
>> 1) bounce the instances that have no partitions assigned (not bounce them
>> all together, but in rolling manner), and see if after the rebalance they
>> are evenly distributed.
>>
>> 2) check if the app.id are set to the same value.
>>
>> Also which version of Kafka are you using?
>>
>>
>> Guozhang
>>
>>
>>
>>
>> On Mon, Dec 14, 2020 at 11:29 AM Navneeth Krishnan <
>> reachnavneeth2@gmail.com>
>> wrote:
>>
>> > Hi All,
>> >
>> > How does kafka streams partition assignment work for sources? I have a
>> > stream application reading from a topic which has 24 partitions. There
>> are
>> > 6 application containers with 4 stream tasks in each container running
>> but
>> > only 2 instances are assigned with partitions and even within the two,
>> one
>> > has 21 partitions and the other has 3 partitions assigned. There is
>> > something wrong in the way the partitions are assigned. Is there a way
>> to
>> > evenly distribute partitions?
>> >
>> > *Group Assignment:*
>> > userapp     user-data               5          1716495057
>> 1716504638
>> >    9581
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
>> > userapp     user-data               13         1735721509
>> 1735730754
>> >    9245
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
>> > userapp     user-data               9          1729975595
>> 1729984597
>> >    9002
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
>> > userapp     user-data               17         1736457935
>> 1736466997
>> >    9062
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
>> > userapp     user-data               0          1917953948
>> 1917963103
>> >    9155
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
>> > userapp     user-data               11         1719082482
>> 1719085979
>> >    3497
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp     user-data               19         1755067810
>> 1755071386
>> >    3576
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp     user-data               7          1729598367
>> 1729601982
>> >    3615
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp     user-data               1          1949943357
>> 1949947252
>> >    3895
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp     user-data               15         1751889619
>> 1751893116
>> >    3497
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp     user-data               3          1752352594
>> 1752356255
>> >    3661
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp     user-data               6          1693612741
>> 1693621514
>> >    8773
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
>> > userapp     user-data               2          1923482321
>> 1923490826
>> >    8505
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
>> > userapp     user-data               14         1677960791
>> 1677969039
>> >    8248
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
>> > userapp     user-data               10         1722133802
>> 1722142388
>> >    8586
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
>> > userapp     user-data               18         1739152927
>> 1739161360
>> >    8433
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
>> > userapp     user-data               20         1715428183
>> 1715431991
>> >    3808
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
>> > userapp     user-data               8          1738854860
>> 1738858541
>> >    3681
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
>> > userapp     user-data               4          1704061352
>> 1704065142
>> >    3790
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
>> > userapp     user-data               12         1743370130
>> 1743373727
>> >    3597
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
>> > userapp     user-data               16         1696027581
>> 1696031399
>> >    3818
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
>> > /172.21.82.140
>> >  userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
>> > userapp     user-data               21         1722643635
>> 1722647766
>> >    4131
>> >
>> >
>> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer-19fdbf9f-fb65-45fb-a020-aa16e9da76c5
>> > /172.21.64.86
>> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer
>> > userapp     user-data               23         1675792619
>> 1675800530
>> >    7911
>> >
>> >
>> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer-e98dae60-89b9-44b0-a366-d533b288fcf8
>> > /172.21.64.86
>> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer
>> > userapp     user-data               22         1735482901
>> 1735490720
>> >    7819
>> >
>> >
>> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer-3553c1a0-def4-453d-a4fb-a6c4f08ad3f2
>> > /172.21.64.86
>> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer
>> >
>> > Regards,
>> > Navneeth
>> >
>>
>>
>> --
>> -- Guozhang
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message