kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruno Cadonna <cado...@apache.org>
Subject Re: Changing Replication Factor
Date Thu, 29 Apr 2021 09:16:10 GMT
Hi Marcus,

1. If you change REPLICATION_FACTOR_CONFIG without resetting the 
application (or deleting the changelog and repartition topics) and 
redeploy the Streams application, the replication factor of the internal 
topics will not change. The replication factor will only change for new 
deployments, i.e., Streams applications with a new application ID or 
Streams applications that where reset. In both cases the internal topics 
will be newly created.

2. Changing the replication factor of a topic directly on the brokers 
should be fine. Kafka Streams should not re-create the internal topics 
and not throw any exceptions.

3. Unfortunately, I do not know the answer to this question. Hopefully 
somebody else can answer it.

I answered your questions to the best of my knowledge. The only way to 
confirm my answers is to test (preferably in a test environment).


Best,
Bruno

On 28.04.21 17:34, Marcus Horsley-Rai wrote:
> Hi All,
> 
> I'm in a sub-optimal situation whereby I have some Kafka Streams apps
> deployed to production, but the default replication factor set on the
> brokers was 1 when they were first deployed.
> As such, any state store changelog topics, and re-partition topics
> therefore have RF 1 also.
> 
> I'm familiar with the bin/kafka-reassign-partitions.sh tool and how to use
> that.
> I've since also found the streams replication.factor
> (StreamsConfig.REPLICATION_FACTOR_CONFIG) setting that can be supplied in
> the streams apps config.
> 
> My questions are:
> 
>     1.  Will simply changing the value of REPLICATION_FACTOR_CONFIG and
>     re-deploying have any effect on already-created internal topics?
>     2. Conversely, should I just change the RF of the internal topics using
>     the re-assign-partitions tool? Is that safe to do whilst the apps are still
>     running?
>     3. (not Streams-specific) If a broker that was the leader of
>     partition(s) for a topic with RF 1 died (i.e. no replicas), and was
>     non-recoverable - would the only way to restore service to those partitions
>     be to delete the topic and re-create?  (with data loss, understandably). I
>     couldn't seem to achieve this using preferred/unclean leader election, nor
>     using the re-assign-partitions tool, since that requires all brokers to be
>     healthy.
> 
> Many thanks in advance for any answers,
> 
> Marcus
> 

Mime
View raw message