kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From emailtokirank@gmail.com <emailtokir...@gmail.com>
Subject Kafka Streams (DSL API) - Any side effects of having more sub topologies?
Date Thu, 13 Jun 2019 16:56:08 GMT

We are using Kafka streams 2.1.1 and leveraging DSL APIs for doing stateful processing (aggregation).
As per our use case we ended up creating ~400 sub topologies. 
Our observation is that based on the number of partitions in the input/source topic, corresponding
stream tasks get created for each of these sub topologies. Wanted to know if having so many
sub topologies is a bad idea OR is there a practical limit on number of sub topologies?

Other observation is that due to more number of sub topologies, Kafka cluster takes more time
to create corresponding internal topics & respond back to kafka streams application resulting
in endless "(Re-)joining group" loop.

We have overcome this by increasing the request & session time out values. But wanted
to be sure if it will have any other side effects.

View raw message