kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: kafka streams replication.factor=3 triples the count of tasks for internal topics
Date Sun, 10 Nov 2019 09:00:12 GMT
The replication factor should not change the number of create tasks.
Those only depend on the number of partitions (and your topology).

If you have two repartition topics I assume that your topology has 3
sub-topologies.

Hence, 2 repartition topic with 68 partitons each (and I assume an input
topic with 68 partitions will result in 189 tasks, 68 task per
sub-topology. Even with a replication factor of one, you should have 189
tasks.


-Matthias



On 11/5/19 2:45 PM, Xiyuan Hu wrote:
> Hi,
> 
> I noticed that by default, replication.factor is 1 for Kafka Streams.
> I increased it to 3 as the document suggested.
> Before, I have 2 repartition topics and each has 63 partitions. After
> increasing to replication 3, those each repartition topic has 63 * 3 =
> 189 tasks.
> 
> Is these behavior expected? I though the replication happened on
> broker side, instead of letting client process the tripled traffic.
> 
> Or is there any other config I missed?
> Thanks for the help!
> 


Mime
View raw message