In addition to the assignment issue I see the state of the application
changes every few minutes from Running to Rebalancing.
Other than the default configs only the below two extra config parameters
are set. Currently there are 10 instances and they are constantly
rebalancing.
fetch-max-wait: 3000
session.timeout.ms: 60000
2020-12-14T23:34:54.715-08:00
[userapp-ee690c43-9457-4c0d-a931-6a3819341e48-StreamThread-4] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:54.715-08:00
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b-StreamThread-3] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:54.715-08:00
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b-StreamThread-3] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:54.721-08:00
[userapp-77c70232-bd1e-4d7f-9543-2d5b6ccf2781-StreamThread-4] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-77c70232-bd1e-4d7f-9543-2d5b6ccf2781] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:54.721-08:00
[userapp-77c70232-bd1e-4d7f-9543-2d5b6ccf2781-StreamThread-4] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:54.733-08:00
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a-StreamThread-4] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a] State transition from
RUNNING to REBALANCING
*2020-12-14T23:34:54.733-08:00*
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a-StreamThread-4] INFO State
transition from RUNNING to REBALANCING
*2020-12-14T23:34:57.898-08:00
*[userapp-edc066cc-2a51-4c92-988c-0b03f1015398-StreamThread-1]
INFO org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-edc066cc-2a51-4c92-988c-0b03f1015398] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:57.898-08:00
[userapp-edc066cc-2a51-4c92-988c-0b03f1015398-StreamThread-1] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:57.899-08:00
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b-StreamThread-4] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:57.899-08:00
[userapp-fb4df2fb-88fe-4e19-8626-d4d695966b7b-StreamThread-4] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:57.899-08:00
[userapp-0f2af06e-e8f2-4dc2-b852-0a6484c7e4e9-StreamThread-3] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-0f2af06e-e8f2-4dc2-b852-0a6484c7e4e9] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:57.899-08:00
[userapp-0f2af06e-e8f2-4dc2-b852-0a6484c7e4e9-StreamThread-3] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:57.900-08:00
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a-StreamThread-4] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:57.900-08:00
[userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a-StreamThread-4] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:57.900-08:00
[userapp-ee690c43-9457-4c0d-a931-6a3819341e48-StreamThread-3] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-ee690c43-9457-4c0d-a931-6a3819341e48] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:57.900-08:00
[userapp-ee690c43-9457-4c0d-a931-6a3819341e48-StreamThread-3] INFO State
transition from RUNNING to REBALANCING
2020-12-14T23:34:57.900-08:00
[userapp-bd83fccf-8cc7-4f80-bd94-42cffd65faf5-StreamThread-2] INFO
org.apache.kafka.streams.KafkaStreams - stream-client
[userapp-bd83fccf-8cc7-4f80-bd94-42cffd65faf5] State transition from
RUNNING to REBALANCING
2020-12-14T23:34:57.900-08:00
[userapp-bd83fccf-8cc7-4f80-bd94-42cffd65faf5-StreamThread-2] INFO State
transition from RUNNING to REBALANCING
Thanks
On Mon, Dec 14, 2020 at 4:07 PM Navneeth Krishnan <reachnavneeth2@gmail.com>
wrote:
> Thanks Guozhang for the suggestion.
>
> We are using kafka 2.3.0 and the app.id is set to the same value.
>
> Bouncing off instances work for a small period of time but when auto
> scaling happens it skews it up again. This is becoming a big bottleneck for
> us since only one node is now trying to handle more than 60% of traffic and
> we are getting into huge lag.
>
> Regards,
> Navneeth
>
> On Mon, Dec 14, 2020 at 2:49 PM Guozhang Wang <wangguoz@gmail.com> wrote:
>
>> Kafka Streams should evenly distribute the partitions, but there are some
>> issues in old versions of Kafka that you may be observing. To verify if it
>> is a transient issue or it is permanently, I'd suggest you try:
>>
>> 1) bounce the instances that have no partitions assigned (not bounce them
>> all together, but in rolling manner), and see if after the rebalance they
>> are evenly distributed.
>>
>> 2) check if the app.id are set to the same value.
>>
>> Also which version of Kafka are you using?
>>
>>
>> Guozhang
>>
>>
>>
>>
>> On Mon, Dec 14, 2020 at 11:29 AM Navneeth Krishnan <
>> reachnavneeth2@gmail.com>
>> wrote:
>>
>> > Hi All,
>> >
>> > How does kafka streams partition assignment work for sources? I have a
>> > stream application reading from a topic which has 24 partitions. There
>> are
>> > 6 application containers with 4 stream tasks in each container running
>> but
>> > only 2 instances are assigned with partitions and even within the two,
>> one
>> > has 21 partitions and the other has 3 partitions assigned. There is
>> > something wrong in the way the partitions are assigned. Is there a way
>> to
>> > evenly distribute partitions?
>> >
>> > *Group Assignment:*
>> > userapp user-data 5 1716495057
>> 1716504638
>> > 9581
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
>> > userapp user-data 13 1735721509
>> 1735730754
>> > 9245
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
>> > userapp user-data 9 1729975595
>> 1729984597
>> > 9002
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
>> > userapp user-data 17 1736457935
>> 1736466997
>> > 9062
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
>> > userapp user-data 0 1917953948
>> 1917963103
>> > 9155
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer
>> > userapp user-data 11 1719082482
>> 1719085979
>> > 3497
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp user-data 19 1755067810
>> 1755071386
>> > 3576
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp user-data 7 1729598367
>> 1729601982
>> > 3615
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp user-data 1 1949943357
>> 1949947252
>> > 3895
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp user-data 15 1751889619
>> 1751893116
>> > 3497
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp user-data 3 1752352594
>> 1752356255
>> > 3661
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer
>> > userapp user-data 6 1693612741
>> 1693621514
>> > 8773
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
>> > userapp user-data 2 1923482321
>> 1923490826
>> > 8505
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
>> > userapp user-data 14 1677960791
>> 1677969039
>> > 8248
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
>> > userapp user-data 10 1722133802
>> 1722142388
>> > 8586
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
>> > userapp user-data 18 1739152927
>> 1739161360
>> > 8433
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer
>> > userapp user-data 20 1715428183
>> 1715431991
>> > 3808
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
>> > userapp user-data 8 1738854860
>> 1738858541
>> > 3681
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
>> > userapp user-data 4 1704061352
>> 1704065142
>> > 3790
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
>> > userapp user-data 12 1743370130
>> 1743373727
>> > 3597
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
>> > userapp user-data 16 1696027581
>> 1696031399
>> > 3818
>> >
>> >
>> userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895
>> > /172.21.82.140
>> > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer
>> > userapp user-data 21 1722643635
>> 1722647766
>> > 4131
>> >
>> >
>> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer-19fdbf9f-fb65-45fb-a020-aa16e9da76c5
>> > /172.21.64.86
>> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer
>> > userapp user-data 23 1675792619
>> 1675800530
>> > 7911
>> >
>> >
>> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer-e98dae60-89b9-44b0-a366-d533b288fcf8
>> > /172.21.64.86
>> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer
>> > userapp user-data 22 1735482901
>> 1735490720
>> > 7819
>> >
>> >
>> userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer-3553c1a0-def4-453d-a4fb-a6c4f08ad3f2
>> > /172.21.64.86
>> > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer
>> >
>> > Regards,
>> > Navneeth
>> >
>>
>>
>> --
>> -- Guozhang
>>
>
|